# Can I think of the level of a hypothesis test as being the probability the null hypothesis is true? [duplicate]

I am trying to understand the level of a test better and was wondering if the level of a hypothesis is essentially equal to the probability that the null hypothesis is true. I have been trying to think of a counter example but haven't been able to confirm nor deny this association. Could anyone be kind enough to shed light on my understanding? Thank you!

• Also think about how to interpret the pvalue if the null hypothesis is false (eg the number of hairs on both sides of sheep are exactly equal). There is nothing stopping you from calculating a p value, but the procedure of accepting/rejecting this type of hypothesis does not seem very meaningful. – Livid Mar 27 '14 at 17:03

(Here be dragons, but I'll try not to mess this up...)

The null hypothesis either is or isn't true. The $p$ value tells you how likely it would be to get the result (a test statistic, and hence the data you collected) you did on the test if the null hypothesis were true. Hence your suggestion to use the level of the test as the probability that the null hypothesis is true is not the correct interpretation - the null is either true or false, nothing in between.

Consider a $t$-test where we are interested scientifically in whether two groups of samples differ in their means for some variable $y$. The null hypothesis in this case is that there is no difference in the means, that the two groups are drawn from the same population (or populations with the same mean). We perform the test and get a $p$ value for the test statistic $t$.

If we don't care (or know, expect) if the difference in means is positive or negative (odd, but what I mean is the second group could have a smaller or larger mean than the first), then we are doing a two-tailed text and the $p$ value is the probability getting an absolute value for the $t$ statistic as large or larger than the observed absolute value of $t$ if the null hypothesis (of no difference) were true. We talk about absolute values of $t$, $|t|$, because large negative values as well as large positive ones are both evidence that the null hypothesis is not true.

(If the alternative hypothesis in the above example was that we expected the mean of group two to be larger (smaller) than that of group one, then we would only consider large positive (negative) values of $t$ as evidence against the null hypothesis respectively.)

In more general terms, you'll often see the $p$ value described as the probability of getting a statistic at least as extreme as the observed statistic, where extreme depends on exactly what the distribution of the test statistic is, what the alternative hypothesis is etc.

That is not correct. It is simply the probability of rejecting the hypothesis if it is true. You are unknowingly mixing up two different conceptions of probability: uncertainty vs. variability.

Roughly, variability requires that probability statements refer to observable statsitics. Uncertainty can refer to unknown, fixed values.

You are trying to attach a probability to a hypothesis. One way to conceretely see how your interpretation is incorrect is to use as follows:

1. Imagine you have a jar of poorly made coins, such that the probabilty of a particular coin landing heads is distributed with 50% having p=0.5, and 50% having p=0.1. This a description of the population of coin probabilities.
2. You pick one of the coins at random and flip it 20 times.
3. You run a hypothesis test for $p=0.5$ and reject at the 0.05 significance level.

Note that the probability of your null hypothesis being true is: $\frac{P(p=0.5)P(Reject|P=0.5)}{p(p=0.5)P(Reject|p=0.5)+p(p=0.1)P(Reject|p=0.1)}=\frac{0.5*.05}{0.5*.05+0.5(1-\beta(p=0.1))}<\frac{.025}{.025+.025} = 50\%$ Where the bound is based on the fact that the power is $\geq$ the Type I rejection rate.

Since the power will vary smoothly between 0 and 100% for this problem, it is not the case that the probability your null is true is equal to the significance.