Why do we compare p-value to significance level in hypothesis testing of mean?

Question

I know that:

p-value = P(observed or more extreme outcome of sample mean | Null hypothesis is true)

But why do we compare it with significance level (alpha) while determining if it is high or low?

How is significance level defined in this context?

Answer 1

The significance level ($\alpha$) is the rate at which you make Type I errors when the null hypothesis is true (or, for composite hypotheses, the maximum rate under the null).

You choose that rate.

Then any test statistic that's more extreme than the one that cuts off $\alpha$ in the tail (i.e. a test statistics more in keeping with the alternative) will cut off a smaller area. That area is the p-value. So when the p-value is small, it means your sample yields a test statistic inside the rejection region.

(the picture is similar for two-tailed tests, but then yellow and green areas occur in both tails)

In order that you actually get that rate of rejection when the null is true, you need to reject that proportion of more extreme cases under the null -- so if your test statistic cuts off a smaller area (green) than the significance level, it's in the region of sample arrangements (in this case, those with unusually large means) that will lead you to reject.

Answer 2

The value of alpha is completely arbitrary. It represents the cut-off that the person who conducted the test has predetermined as significantly low as to warrant betting against the null hypothesis.

Conventionally this is often 0.05 which is just derived from what Fischer thought was reasonable. However it is perfectly possible to have different levels.

Ultimately, it is up to the judgement of the person carrying out the test.

Answer 3

I lightly rewrite this already outstanding /r/eli5 comment, to simplify it further. I changed "New Yorker to "Utahn" as the latter's shorter.

---

Suppose you want to show that, say, Texans eat more than Utahns do. What you're really trying to prove is that Texans do not eat the same or less amount as Utahns do. This statement ("Texans and Utahns eat the same amount") is something called your "null hypothesis". Hypothesis testing has the goal of disproving the null hypothesis to prove what you're trying to show.

The idea of statistical testing is to say "well, assuming that Texans and Utahns did eat the same amount, how likely would we get the data we did? The chance of getting the data you got if, in fact, they did eat the same amount is called a p-value. For instance, if we say that p = 0.05, we mean that if Texans and Utahns ate the same amount, there'd be a 1/20 chance to observe the kinds of results we did observe. The lower the p-value, the less likely your null hypothesis is true, and the more confidence you have that, in fact, Texans do eat more.

Significance is the lowest p-value you'll accept as "strong enough" evidence. Lower significance thresholds decrease your chances of a false positive (i.e., finding that Texans eat more when in fact they don't), but increase your chances of a false negative (concluding that you don't know that Texans eat more, when in fact they do). Usually 5% is the weakest significance anyone takes seriously, but for situations where there's extreme cost to a false positive, you may choose a much lower number like 0.1%.