Say we have a null hypothesis, $H_0$ that there is no difference in the proportions between two arms of an experiment. You power the test to alpha and beta levels such that your $H_1$ MDE is 1% relative.

After the test, we fail to reject the null. We find only a 0.1% difference between treatment and control and our p value is 0.7.

My question is: Can we calculate the probability of observing such a small effect assuming alternative is true? So I want to be able to say something like: Given how small the effect is, there's a 99% chance of observing such a small effect assuming the true value of the treatment was really 1%. Can I do this?

So in this sense, we have our original null hypothesis, then we have a probabilistic statement about the treatment observations against a fixed mean.

In my case, where a normal approximation to binomial is fine (N > 10k), it seems reasonable to me to make this kind of probabilistic statement just by looking at how many z-scores away the observed mean is from the fixed mean.

  • $\begingroup$ Yes, you can. A likelihood function that will show the relative probabilities of obtaining data like those you got as a function of the true effect size. You don't know the 'true' effect size, so the likelihood function is usually a better representation of what happened than any single likelihood value or ratio of likelihoods for arbitrary effect sizes. $\endgroup$ Sep 15, 2023 at 5:12
  • 1
    $\begingroup$ 1. Note that the p-value is not "the probability of observing the treatment" given $H_0$. The probability of "observing the treatment" for a continuous outcome is 0. $\,$ 2. If you reframe the question to reflect the thing you calculated in the p-value then under $H_1$, it depends on the specific alternative. If you have a simple (equality) null and a compound (inequality) alternative, then you would need to choose which of the specific alternatives you mean; it would be different for each case. $\endgroup$
    – Glen_b
    Sep 15, 2023 at 5:13
  • $\begingroup$ @Glen_b, Yes, I tried to avoid framing the question as a "two null hypotheses" problem and instead just framed this as a z-score question against a known mean. in this case, we anchor on a mean that we simply state is true. to your point about the continuous nature of the value, then we could calculate the probabilty of observing the treatment at least as far from the stated mean given that ground truth. $\endgroup$ Sep 15, 2023 at 5:39
  • $\begingroup$ How can I make my question more clear? I don't fully understand what you are trying to say in response. @Glen_b $\endgroup$ Sep 15, 2023 at 5:40
  • $\begingroup$ @MichaelLew Would I then do a LRT given the sup for the observed data in the denominator and the sup for the MDE assumed mean in the numerator? that starts to just feel like another hypothesis test. $\endgroup$ Sep 15, 2023 at 5:43

1 Answer 1


Yes we can, and in fact, this calculation is just a variation on the power function. Suppose you have a test using the parameter $\theta \in \Theta$ and with significance level $\alpha$. Then the negated-power function is:

$$\beta_\alpha(\theta) = \mathbb{P}(\text{Accept } H_0 \text{ in test at level } \alpha | \theta) = \mathbb{P}(p(\mathbf{X}) \geqslant \alpha | \theta) \quad \quad \quad \text{for all } \theta \in \Theta.$$

You observed a p-value of $p=0.70$ in your test so the probability of interest to you would just be negated-power function for a test using significance level $\alpha = 0.70$.

$$\beta_{0.70}(\theta) = \mathbb{P}(p(\mathbf{X}) \geqslant 0.70 | \theta).$$

If you evaluate this at a parameter value in the alternative space then you will obtain the probability of "observing such a small effect" assuming a parameter value where the alternative is true. (If you want to condition on the entire alternative space rather than a specific parameter value in that space, you would then need to specify a prior distribution for the parameter over that space.)

  • 2
    $\begingroup$ This is a beautiful reply. But I think I am simply not as good at math and statistics are you are. In practice, assuming my parameter is the $\mu$ of a normal distribution that approximates a binomial distribution, it seems I can simply use the pivotal z-score resulting from the observed treatment value I want to assess against the fixed mean. from there, we conduct the test as usual. as you say, this is a variation on how the test was powered originally (using just the power function). $\endgroup$ Sep 15, 2023 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.