7
$\begingroup$

I teach introductory statistics to undergraduates and they are often confused with hypothesis testing. In particular, while the rule is

we reject the null hypothesis at significance level $\alpha$ when p value is less than $\alpha$

they many times interpret it the opposite. Say, if p value is 0.04, they say "we reject at 1% but not at 5%".

On one level, it is about the deeper understanding, which might be my fault as a teacher. But on another level (given that we are not always engaging with the deeper side of things), perhaps a cool mnemonic tip would help them with correct interpretation

Do you have a cool, undergraduate-level tip about how to correctly interpret p values vs significance level $\alpha$? I haven't come across any such tip.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Interesting, students & academics are often confused about p-values. but this is a new one to me $\endgroup$
    – innisfree
    Feb 24, 2021 at 5:37
  • 1
    $\begingroup$ The more you emphasise P-values, the less attention is needed for whether the hypothesis is rejected. $\endgroup$
    – Nick Cox
    May 29 at 11:10
  • $\begingroup$ I see exactly this one too often. I still have the hope that most of such answers come from students that say that they are not interested in statistics $\endgroup$
    – Ute
    May 29 at 12:03

4 Answers 4

Reset to default
4
$\begingroup$

This surely will not top the list of possible "cool undergraduate-level tips", but simply recalling the definition of a p-value might be helpful (quoted from Wikipedia):

The probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct.

So the smaller the probability, the smaller significance level at which we are willing to reject.

Improve this answer
$\endgroup$
1
  • $\begingroup$ +1 recalling and applying definitions for terms is an excellent skill to build in undergraduates and students in science. $\endgroup$
    – AdamO
    May 31 at 13:11
2
$\begingroup$

The standard mnemonic for remembering how to make a conclusion in a hypothesis test is:

If p is low, the null must go!

As to why this is the case, the best explanation of a classical hypothesis test is that it is the inductive anologue of a proof by contradiction. In a proof by contradiction we begin with a null hypothesis, show that this leads logically to a contradiction, and therefore reject the initial premise that the null is true. In a classical hypothesis test, we begin with a null hypothesis, show that this leads to a highly implausible result in favour of the alternative (so not quite a deductive contradiction, but close), and therefore reject the initial premise that the null is true. The p-value in this test is the probability of a result at least as conducive to the alternative hypothesis, assuming the null is true (see formal explanation here). If this is low then it means that something very implausible happened (under the assumption that the null is true) which gives the "contradiction" in the "inductive proof by contradiction".

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the mnemonic - if anyone knows a Danish one, please post here, too :-) $\endgroup$
    – Ute
    May 29 at 10:57
  • 1
    $\begingroup$ It's not an answer but I sometimes emphasise that there is nothing unusual about a figure of merit where low means good, as witness unemployment, inflation, goals let in by your favourite team, number of criminal convictions. $\endgroup$
    – Nick Cox
    May 29 at 11:12
  • $\begingroup$ "good" is subjective. If you want to defend the null hypothesis for whatever reason, then large p-values are good... $\endgroup$
    – Ute
    May 29 at 12:17
1
$\begingroup$

Fisher is said to have given the interpretion of $p$-values as a "measure of surprise", given you believe in the null hypothesis. This may actually be confusing, since low $p$-value then indicates strong surprise.

Instead, we can introduce $p$-values as "measure of compatibility with the null". (suggested by Christian Hennig) Then: low p = low compatibility

In Cox And Hinkley's 1974 text, they use the p-value "as a measure of the consistency of the data with the null hypothesis" (p.66). Earlier, Cox (1958) described a significance test as "concerned with the extent to which the data are consistent with the null hypothesis".(p. 362)

Improve this answer
$\endgroup$
15
  • $\begingroup$ Should "trust" be replaced by "belief" or something else (non native speaker) $\endgroup$
    – Ute
    May 29 at 10:59
  • 4
    $\begingroup$ Terms with psychological overtunes already hinder more than they help, Confidence and significance already have done much harm because confidence in a statistical sense is nothing do with what the researcher feels or should feel and significance is just a vague word that could mean almost anything. There is no easy solution here because completely new words or new technical jargon e.g. calling a confidence interval a Neyman interval would both be obnoxious in different ways. $\endgroup$
    – Nick Cox
    May 29 at 11:09
  • 3
    $\begingroup$ Never ever should we trust in a null hypothesis (which implies all model assumptions), regardless of the p-value. Neither should we believe it. Re "measure of agreement" - what about "compatibility" (as used in the ASA recommendations about p-values)? $\endgroup$
    – Christian Hennig
    May 29 at 14:27
  • 1
    $\begingroup$ I think a safe way to view the p-value as measuring the "(statistical) consistency of the data with the null hypothesis". Lower value means less consistency. With this view it is hard to make the common misinterpretations. In particular, it makes it clear that p-value is about the absolute plausibilty of the null, and is not a measure of relative plausibility of the null. $\endgroup$
    – Graham Bornholt
    May 29 at 20:58
  • 2
    $\begingroup$ The students should be exposed to the much more intuitive and actionable Bayesian approach IMHO. But if only teaching the classical frequentist approach I'd do everything possible to not teach hypothesis tests but rather them them interpretations as the following. "The ... test resulted in p=0.04 representing modest evidence against the supposition that there is no effect... The test resulted in p=0.11 indicating little evidence against the supposition that the treatment is ignorable...".For both of these examples the phrase "assuming model assumptions hold" needs to be appended. $\endgroup$
    – Frank Harrell
    May 31 at 12:41
1
$\begingroup$

I've found that some of my students are helped by thinking of the p-value as a percentile. They are familiar with the concepts of being in the top 10% of a class by GPAs, or "among the 1%" in terms of wealth.

So for your example, a p-value of 0.04 means "Our observed value of the test statistic $T$ was among the top 4% possible values of $T$ under $H_0$ that are least like $H_0$ and most like $H_A$."

In other words, "Our observed test statistic was among the top 5% most un-$H_0$-like values, but not among the top 1%."

Improve this answer
$\endgroup$
3
  • $\begingroup$ +1 You seem to be tapping into the psychological literature that suggests people don't think accurately about probabilities but can do very well when the same thing is cast in concrete terms of "out of every 100, ...". To take this to its logical conclusion, then, you might consider stating "This value of $T$ is evidence against $H_0$ in the sense that when $H_0$ is the case, we expect only four out of every 100 experiments to exhibit a value of $T$ that counts against $H_0$ as strongly as the observed value does, while there is some $H_A$ more likely to produce such a strong value of $T.$" $\endgroup$
    – whuber ♦
    May 31 at 19:40
  • $\begingroup$ @whuber Thanks! I'm somewhat aware of the psych literature you mention (work by Tversky & Kahneman and others), but I don't know it very deeply. I suspect that my suggested wording ("among the top 4%...") taps into slightly different intuition than your suggested wording ("only 4 out of every 100..."), but I would be really curious to know if they have been compared in the psych literature. $\endgroup$
    – civilstat
    May 31 at 20:27
  • $\begingroup$ I believe Gerd Gigerenzer has written about this. $\endgroup$
    – whuber ♦
    May 31 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.