How to simplify the understanding behind Hypothesis Testing? [duplicate]

Question

Quote recently, I have trying to understand this confusing idea of Hypothesis testing. Although I am pretty much clear with the way it works, that is:

• Choosing a test statistic
• Then choosing a Null Hypothesis (H0) & an alternative hypothesis (H1)
• Finding out the P-Value
• If the p-value is greater than the significance level (assuming 0.05 for this question), then accepting H0, otherwise rejecting it

But I am still highly confused with the last step. Let me throw some light on my confusion with 2 examples.

The 1st example is to check whether a coin is biased towards heads or not. So, we design an experiment in which we flip the coin 5 times, and the number of heads we obtain is the test-statistic and the coin is unbiased is our Null Hypothesis. Now, let's say we flip the coin 5 times, and we obtain 5 heads. So, P(5 heads | The coin is unbiased) = 1/32 ~ 0.03 (< 0.05), and hence we reject H0. The intuition behind it is pretty simple as well since the probability of obtaining 5 heads in 5 tosses, from an unbiased coin is very less, hence we concluded that the coin is biased.

Now, comes the 2nd example, which puts me into great confusion. In this example, we need to determine if the heights of students from 2 different classes C1 & C2 follow the same distribution or not. So, we choose the test statistic (X) as the difference between the means of the heights u1 & u2, and we choose the null hypothesis as there being no difference in the means. Now, let's say that we performed an experiment and found out X to be 10cm. Now, we try to determine the p-value, that is P(X >= 10cm | H0), and let's say we obtain it to be 0.01 (< 0.05), and hence we reject the null hypothesis.

Now, here is my doubt, if the p-value mentioned above is very less, then shouldn't it mean that there is no difference in the heights of students of C1 and C2? In simple words, if the p-value is 0.01, then shouldn't it mean that it is very less probable that the difference in the means of C1 and C2 is big, i.e. no difference in the heights of C1 and C2 (exactly the same as our Null Hypothesis), but it happens to be completely opposite?

P.S. - This is my first question on any discussion forums, so let me know if I am missing some information that I need to add. Thanks in advance :)

Answer 1

In short, the p-value always gives you the probability of the found result assuming that the H0 hypothesis is true, i.e. that there is no difference between the two samples.

So in your second example, the p-value of 0.01 tells you that - assuming both groups indeed do follow the same distribution - the measured difference only occurs in 1% of all cases. If the p-value now gets smaller, say, 0.001, the result becomes even more unlikely under the null hypothesis and the more unlikely it becomes (the smaller the p-value), the more we are inclined to reject the null hypothesis.

The low p-value indeed says that a large difference is unlikely. However, you did in fact observe this (large) difference and this observation is very unlikely when H0 holds. Thus, you reject the null hypothesis and assume the alternative hypothesis - that the samples follow different distributions - to be true.