What is the probability that a hypothesis test fails?

Question

If $X\sim P$, given some other distribution $Q\gg P$ what is known about $\mathbb{P}(P(X)< Q(X))$, i.e. the probability the outcome was more likely to have come from $Q$?

In particular are there any bounds on the quantity in terms of statistical distances?

Answer 1

For general PMFs $P,Q$ with $X\sim P$ then nothing interesting can be said about $\mathbb{P}(P(X)<Q(X))$ for all the metrics I know about:

Imagine $P(x) = \frac{1}{n}$ for $x\in[1:n]$, 0 otherwise, and $Q(x) = \frac{1-\varepsilon}{n-1}$ for $x\in [1:n-1],$ $Q(n)=\varepsilon$ and 0 otherwise. For $X\sim P$, then $\mathbb{P}(P(X)<Q(X))=1-\varepsilon-\frac{1}{n}$, even though all these distance metrics go to 0:

• $\|P-Q\|_1\approx 2/n$,
• $\|P-Q\|_2\approx \frac{1}{n\cdot (n-1)}$ and
• $D(P\|Q)\leq \log(\frac{n-1}{n})-\frac{1}{n}\log(\varepsilon n)$.

With slight adjustment, the following holds in general (follows easily by expanding it into an integral of an indicator function):

\begin{align} \mathbb{P}(P(X)<c\cdot Q(X)) \leq c. \end{align}