If you're happy to assume each count follows a Poisson distribution (with its own mean under the alternative hypothesis; with a common mean under the null) there's no problem—it's just that you can't check that assumption without replicates. Overdispersion can be quite common with count data.
An exact test given counts $x$ & $y$ is straightforward because the overall total of counts, $n$ is ancillary; conditioning on it gives $X \sim \mathrm{Bin}\left(\frac{1}{2},n\right)$ as the distribution of your test statistic under the null.