# Intermediate effect sizes in hypothesis tests

Background: Consider a null hypothesis test. Denote $T$ the indicator r.v. for the outcome of the test, with $T=1$ meaning the null hypothesis was rejected (positive test result). Denote $H_0$ the event that the null hypothesis is true and $H_1$ the event that the alternative hypothesis is true, which is the complementary event to $H_0$, thus $P(H_0) = 1 - P(H_1)$. Typically, we control $P(T=1|H_0)$, sometimes called the "type I error rate". What we are actually interested in are $P(H_1|T=1)$ and $P(H_0|T=0)$, which, using jargon from diagnostic tests, are called the "positive predictive value" and "negative predictive value", respectively. Their counter-probabilities $P(H_0|T=1)$ and $P(H_1|T=0)$ are known as "crucial type I error rate" and "crucial type II error rate", respectively [1]. The first, $P(H_0|T=1)$, is also known as the "false discovery rate" [2] or "false positive risk" [3].

By Bayes' theorem, we obtain the odds for the positive predictive value, also known as the "posterior odds" for $H_1$: $$\frac{P(H_1|T=1)}{P(H_0|T=1)} = \frac{P(T=1|H_1)}{P(T=1|H_0)} \cdot \frac{P(H_1)}{P(H_0)}$$

The second factor, $\frac{P(H_1)}{P(H_0)}$, also known as the "prior odds" for $H_1$, is pretty much down to subjectivity. Assuming a value of more than $1$ is often considered as non-justifiable. Using exactly $1$ means that a priori we do not favor $H_0$ over $H_1$ or vice versa.

Let us move our attention to the first factor, namely $\frac{P(T=1|H_1)}{P(T=1|H_0)}$, in diagnostic test jargon known as "likelihood ratio". It is a property of the test, independent of $P(H_0)$ and $P(H_1)$. A lower bound on it gives an idea of how much information is in a positive test result. See, e.g., [3] for interesting ideas what can be done with such a lower bound ("reverse Bayesian argument" for instance).

The denominator, which is the type I error rate, is not a problem, since it typically is under our control; at least we have an upper bound on it.

Problem statement: The enumerator, $P(T=1|H_1)$, is more of a problem. It is known as the "power" of the test. Let's assume that $H_0$ says that the effect size is between $0$ and some $d_1 > 0$. Then $H_1$ means that the effect size is at least $d_1$. Say that we have $P(T=1|H_0) \leq \alpha$ for some (small) alpha, e.g., $\alpha = 0.05$. In order to obtain a reasonable lower-bound on $P(T=1|H_1)$, to the best of my knowledge, we have to specify a second effect size, $d_2 > d_1$, and modify $H_1$ so that it means that the effect size is at least $d_2$. But then, without further assumptions, $H_0$ and $H_1$ are not complementary anymore, and Bayes' theorem cannot be used like it was used above. One way is to make the additional assumption that effect size is either below $d_1$ or above $d_2$, so we a priori rule out effect sizes between $d_1$ and $d_2$.

This is done in [2] and [3] and stated clearly there, and it appears to me that this assumption is appropriate for the purposes of that study. However, regarding [1], I am not so sure. Indeed, a priori ruling out a certain intermediate effect size appears unrealistic in general.

I am aware of that there exist numerous approaches that specify a probability distribution on the effect size, e.g., [4]. However, I was looking for something simpler than that. So I came up with the following.

Approach: Denote $0 < d_1 < d_2$, and call an effect of size in $[0,d_1)$ a "negligible" effect, in $[d_1,d_2)$ it is an "intermediate" effect, and in $[d_2, \infty)$ we have a "substantial" effect. The null hypothesis $H_0$ is that we have a negligible effect, the alternative hypothesis $H_1$ is that we have an intermediate or a substantial effect. Denote $H_1^-$ the event of having an intermediate effect and $H_1^+$ the event of having a substantial effect, so $H_1 = H_1^- \cup H_1^+$ and $H_1^- \cap H_1^+ = \emptyset$. We will in general not get a reasonable lower bound on $P(T=1|H_1^-)$; in many cases we will be allowed to assume $P(T=1|H_1^-) \geq P(T=1|H_0)$, but this does not say much. However, by appropriate choice of $d_2$ and sample size, we can get a reasonable lower bound on $P(T=1|H_1^+)$, that is, on the probability of a positive test given a substantial effect size. The latter can be done by common power analysis. Say that we have $P(T=1|H_0) \leq \alpha$ and $P(T=1|H_1^+) \geq \eta$, where typical values could be $\alpha = 0.05$ and $\eta = 0.80$.

By Bayes' theorem, we obtain a lower bound on the odds for the positive predictive value: $$\frac{P(H_1|T=1)}{P(H_0|T=1)} \geq \frac{P(T=1|H_1^+)}{P(T=1|H_0)} \cdot \frac{P(H_1^+)}{P(H_0)} \geq \frac{\eta}{\alpha} \cdot \frac{P(H_1^+)}{P(H_0)}$$

This is similar to before, but instead of the prior odds $\frac{P(H_1)}{P(H_0)}$, we now got $\frac{P(H_1^+)}{P(H_0)}$. If we assign equal prior probabilities to $H_0$, $H_1^-$, and $H_1^+$ (namely $\frac{1}{3}$ each), we obtain the same lower bound as with the original approach where we assign equal probabilities to $H_0$ and $H_1$. Perhaps, this is the justification in [1] and similar publications for not considering intermediate effects.

However, assigning $\frac{1}{3}$ to each of the three takes away probability from $H_0$, compared to the original approach where $P(H_0)=\frac{1}{2}$. So maybe more honest prior probabilities would be $P(H_0) = \frac{1}{2}$ and $P(H_1^-)=P(H_1^+)=\frac{1}{4}$, reducing the lower bound on the posterior odds for $H_1$ by a factor of $2$.

It becomes easier when the parameter space is bounded. Say that our observation is distributed like $\mathrm{Bin}(n,p)$, where $n$ is known and $p$ not. The task is to make a statement about $p$. Let $0 < p_1 < p_2 < 1$ and define a negligible effect as $p < p_1$, an intermediate one as $p_1 \leq p < p_2$, and a substantial one as $p_2 \leq p$. Then we could take the prior probabilities to be $P(H_0) = p_1$, $P(H_1^-) = p_2 - p_1$, and $P(H_1^+) = 1 - p_2$. (In this case, a more detailed analysis where we assume a uniform prior on $p$ would also be relatively easy.)

Question 1: Can you think of any reason why in the literature, e.g., in [1], it appears to be silently assumed that intermediate effects do not occur? Am I missing something?

Question 2: What do you think of the approach of dividing effect sizes into negligible, intermediate, and substantial? Would this give researchers in the applied sciences a way of more concretely think about the expressiveness of hypothesis tests for their studies, without having to go all the way to prior probability distributions on the effect size (although the latter would not be too difficult for the case of a $\mathrm{Bin}(n,p)$ observation as stated above)? Defining $H_0$ as a negligible effect instead of a point hypothesis is not new, of course, see e.g. [5, Chapter 3]. But explicitly assigning prior probability mass to intermediate effect sizes has, to the best of my knowledge, not been covered in such a direct and simple way before.

References:

[1] O’Brien and Castelloe, "Sample-Size Analysis for Traditional Hypothesis Testing: Concepts and Issues", Chapter 10 in "Pharmaceutical Statistics Using SAS", 2007

[2] Colquhoun, "An investigation of the false discovery rate and the misinterpretation of p-values", Royal Society Open Science, 2014, http://rsos.royalsocietypublishing.org/content/1/3/140216

[3] Colquhoun, "The Reproducibility Of Research And The Misinterpretation Of P Values", 2017, http://www.biorxiv.org/content/early/2017/08/07/144337

[4] Rouder, et al., "Bayesian t tests for accepting and rejecting the null hypothesis", 2009, doi:10.3758/PBR.16.2.225

[5] Murphy, Myors, Wolach, "Statistical Power Analysis", 2014

Addendum: The famous article "Why Most Published Research Findings Are False" (2005) makes it rather explicit that it restricts to certain effect sizes: "Let us also consider, for computational simplicity, circumscribed fields where either there is only one true relationship (among many that can be hypothesized) or the power is similar to find any of the several existing true relationships."