I have N random samples $x_i$ from an unknown probability distribution $A$, and 1 random sample $y$ from another unknown distribution $B$. The distributions can be assumed to be continuous and well-behaved. The null hypothesis $H_0$ is that $A$ and $B$ are the same distribution. I am interested in testing whether $y$ is large enough to conclude that $H_0$ is unlikely. Based on the above criterion, I believe the value I want to calculate is $$p = P[X \geq y | H_0]$$ namely, the probability of randomly drawing from $A$ a value that is at least as large as $y$.

I have tried:

  • Defining a bernoulli random variable $z_i = x_i\geq y$, and estimating $p$ from the resulting binomial distribution. It works fine, but it does not take account of the magnitudes of $x$ and $y$, only their rank, so maybe it is possible to do better?
  • Upper-bounding $p$ using sample Chebyshev's inequality. Again, it works, but seems quite conservative (needs $y$ to be like 15$\sigma$ away from $x$ to achieve $p\leq 0.01$)

What other options exist? Is it possible to improve the situation by some small additional knowledge about $A$ (e.g. unimodality)?

  • $\begingroup$ Ordinarily a p-value is based on a statistic, but your probability expression is not a statistic. Thus, it looks like you need to ask about both a suitable statistic and how to use it to conduct this test. Typically the test is conducted by erecting a prediction limit for $Y$ based on the $X_i.$ $\endgroup$
    – whuber
    Dec 9 '19 at 17:56
  • $\begingroup$ Can't I use the r.v. $X$ itself as a test statistic? Also, I think I have found a possible answer to my second question here stats.stackexchange.com/questions/82419/… $\endgroup$ Dec 9 '19 at 18:23
  • $\begingroup$ If you only have one observation for $Y$, your sample size is effectively $1$ even if you know the theoretical distribution for $X$. Consequently, the central limit theorem can't be relied on. $\endgroup$
    – jbowman
    Dec 9 '19 at 18:44
  • $\begingroup$ @jbowman I don't follow. If I know the theoretical distribution for X, I can evaluate how likely is it that a single observation Y came from that distribution, can't I? I only need CLT to determine the distribution of X, for which I have multiple observations $\endgroup$ Dec 9 '19 at 19:23
  • $\begingroup$ You never observe the random variable: all you have is a realization. In particular, you cannot ever directly observe any probability associated with a random variable. $\endgroup$
    – whuber
    Dec 9 '19 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.