# What is the probability that a hypothesis test fails?

If $X\sim P$, given some other distribution $Q\gg P$ what is known about $\mathbb{P}(P(X)< Q(X))$, i.e. the probability the outcome was more likely to have come from $Q$?

In particular are there any bounds on the quantity in terms of statistical distances?

• What does the double greater than sign mean? Apr 30, 2017 at 22:16
• $P\ll Q$ means $P$ is absolutely continuous with respect to $Q$, which means that "all the events $Q$ says have probability 0, $P$ also says have probability 0." It's so that the ratio $P(X)/Q(X)$ makes sense. Apr 30, 2017 at 22:30
• The question in the title and the question in the body seem like pretty different questions. Also, I'm not sure what the notation $P(X)$ or $Q(X)$ mean; I'm not even sure whether $P$ and $Q$ are supposed to be PDFs, CDFs, probability measures, or something else. May 3, 2017 at 16:27
• The probability I am asking about is precisely "given $X$ came from $P$, what is the probability the outcome is more likely to have come from $Q$" which due to Neyman-Pearson is the most powerful hypothesis test for some significance level. The question only makes sense as stated if $P,Q$ are pmfs on finite alphabets. May 3, 2017 at 17:26

For general PMFs $P,Q$ with $X\sim P$ then nothing interesting can be said about $\mathbb{P}(P(X)<Q(X))$ for all the metrics I know about:
Imagine $P(x) = \frac{1}{n}$ for $x\in[1:n]$, 0 otherwise, and $Q(x) = \frac{1-\varepsilon}{n-1}$ for $x\in [1:n-1],$ $Q(n)=\varepsilon$ and 0 otherwise. For $X\sim P$, then $\mathbb{P}(P(X)<Q(X))=1-\varepsilon-\frac{1}{n}$, even though all these distance metrics go to 0:
• $\|P-Q\|_1\approx 2/n$,
• $\|P-Q\|_2\approx \frac{1}{n\cdot (n-1)}$ and
• $D(P\|Q)\leq \log(\frac{n-1}{n})-\frac{1}{n}\log(\varepsilon n)$.