# When failing to reject a null hypothesis can we also calculate the probability of observing the treatment assuming the alternative is true?

Say we have a null hypothesis, $$H_0$$ that there is no difference in the proportions between two arms of an experiment. You power the test to alpha and beta levels such that your $$H_1$$ MDE is 1% relative.

After the test, we fail to reject the null. We find only a 0.1% difference between treatment and control and our p value is 0.7.

My question is: Can we calculate the probability of observing such a small effect assuming alternative is true? So I want to be able to say something like: Given how small the effect is, there's a 99% chance of observing such a small effect assuming the true value of the treatment was really 1%. Can I do this?

So in this sense, we have our original null hypothesis, then we have a probabilistic statement about the treatment observations against a fixed mean.

In my case, where a normal approximation to binomial is fine (N > 10k), it seems reasonable to me to make this kind of probabilistic statement just by looking at how many z-scores away the observed mean is from the fixed mean.

• Yes, you can. A likelihood function that will show the relative probabilities of obtaining data like those you got as a function of the true effect size. You don't know the 'true' effect size, so the likelihood function is usually a better representation of what happened than any single likelihood value or ratio of likelihoods for arbitrary effect sizes. Sep 15, 2023 at 5:12
• 1. Note that the p-value is not "the probability of observing the treatment" given $H_0$. The probability of "observing the treatment" for a continuous outcome is 0. $\,$ 2. If you reframe the question to reflect the thing you calculated in the p-value then under $H_1$, it depends on the specific alternative. If you have a simple (equality) null and a compound (inequality) alternative, then you would need to choose which of the specific alternatives you mean; it would be different for each case. Sep 15, 2023 at 5:13
• @Glen_b, Yes, I tried to avoid framing the question as a "two null hypotheses" problem and instead just framed this as a z-score question against a known mean. in this case, we anchor on a mean that we simply state is true. to your point about the continuous nature of the value, then we could calculate the probabilty of observing the treatment at least as far from the stated mean given that ground truth. Sep 15, 2023 at 5:39
• How can I make my question more clear? I don't fully understand what you are trying to say in response. @Glen_b Sep 15, 2023 at 5:40
• @MichaelLew Would I then do a LRT given the sup for the observed data in the denominator and the sup for the MDE assumed mean in the numerator? that starts to just feel like another hypothesis test. Sep 15, 2023 at 5:43

Yes we can, and in fact, this calculation is just a variation on the power function. Suppose you have a test using the parameter $$\theta \in \Theta$$ and with significance level $$\alpha$$. Then the negated-power function is:
$$\beta_\alpha(\theta) = \mathbb{P}(\text{Accept } H_0 \text{ in test at level } \alpha | \theta) = \mathbb{P}(p(\mathbf{X}) \geqslant \alpha | \theta) \quad \quad \quad \text{for all } \theta \in \Theta.$$
You observed a p-value of $$p=0.70$$ in your test so the probability of interest to you would just be negated-power function for a test using significance level $$\alpha = 0.70$$.
$$\beta_{0.70}(\theta) = \mathbb{P}(p(\mathbf{X}) \geqslant 0.70 | \theta).$$
• This is a beautiful reply. But I think I am simply not as good at math and statistics are you are. In practice, assuming my parameter is the $\mu$ of a normal distribution that approximates a binomial distribution, it seems I can simply use the pivotal z-score resulting from the observed treatment value I want to assess against the fixed mean. from there, we conduct the test as usual. as you say, this is a variation on how the test was powered originally (using just the power function). Sep 15, 2023 at 6:13