If it is 1804 Birmingham, it MUST have a Duty Mark - it is not small enough to be exempt, and has been assayed. Go to this site -
http://www.925-1000.com/dlBirmingham.html - at the top you will see an example of Birmingham marks for 1801 - your piece should also have the mark on the far left - the Duty Mark. Has it been "rubbed" out by polishing?
While I stand to be corrected, I still doubt that a piece of 1804 silver, otherwise fully stamped by the assay office, would not have been stamped with a duty mark.
Waylander