I have N random samples $x_i$ from an unknown probability distribution $A$, and 1 random sample $y$ from another unknown distribution $B$. The distributions can be assumed to be continuous and well-behaved. The null hypothesis $H_0$ is that $A$ and $B$ are the same distribution. I am interested in testing whether $y$ is large enough to conclude that $H_0$ is unlikely. Based on the above criterion, I believe the value I want to calculate is $$p = P[X \geq y | H_0]$$ namely, the probability of randomly drawing from $A$ a value that is at least as large as $y$.
I have tried:
- Defining a bernoulli random variable $z_i = x_i\geq y$, and estimating $p$ from the resulting binomial distribution. It works fine, but it does not take account of the magnitudes of $x$ and $y$, only their rank, so maybe it is possible to do better?
- Upper-bounding $p$ using sample Chebyshev's inequality. Again, it works, but seems quite conservative (needs $y$ to be like 15$\sigma$ away from $x$ to achieve $p\leq 0.01$)
What other options exist? Is it possible to improve the situation by some small additional knowledge about $A$ (e.g. unimodality)?