0
$\begingroup$

I have been doing statistics as part of my job for a few years now. But now I have to try and explain the logic of Null Hypothesis Significance testing to a statistically naive audience.

Say I have the result of an unpaired t-test: t=0.01, p=0.999

I can conclude that there is no difference between groups but why can't I use this as evidence that the groups are the same or nearly the same?

Put another way, why does rejecting the null hypothesis allow us to make a conclusion but failing to reject it does not?

Why are a finding of equivalence and failing to find a difference not the same? (this is more a semantic question)

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ What did you estimate the effect to be, zero or something other than zero? $\endgroup$
    – Dave
    Commented Mar 23, 2022 at 2:45
  • 1
    $\begingroup$ In other words, you estimated the effect to be something other than zero, right? $\endgroup$
    – Dave
    Commented Mar 23, 2022 at 2:53
  • 1
    $\begingroup$ You estimated the effect to be something other than zero. Why, then, do you want to ignore your estimate and claim the effect to be zero? $\endgroup$
    – Dave
    Commented Mar 23, 2022 at 2:55
  • 2
    $\begingroup$ Because a high p-value could be evidence of a very small sample size. Therefore accepting the null could be mistaken for poor data. If you want to report confidence in an effect close to zero you should focus more on the standard errors than the pvalue. $\endgroup$
    – ecnmetrician
    Commented Mar 23, 2022 at 3:07
  • 3
    $\begingroup$ No. You might have a gigantic confidence interval, making it plausible that the groups are quite different, even if you had the rotten luck of not catching a big difference. // If you want to show that two groups are super close, look into equivalence testing, but be prepared to define what “near-identical” means in terms of effect size. $\endgroup$
    – Dave
    Commented Mar 23, 2022 at 3:08

1 Answer 1

Reset to default
1
$\begingroup$

I can conclude that there is no difference between groups

Not quite. A more precise conclusion would be that the data are consistent with there being no difference. However, we can't know for sure that there is no difference. The difference could be too small to detect reliably with the given sample size for example.

why does rejecting the null hypothesis allow us to make a conclusion but failing to reject it does not?

You have to understand that the two types of "conclusions" are very different. When we reject the null, we say "It is extremely unlikely that we observe these data if there truly were no difference" and hence conclude/decide to believe that there is a difference. But we don't make a precise statement about the difference; the difference requires estimation, and there is uncertainty in that estimate.

"Accepting the null" or making a precise statement that the difference is exactly 0 is just too precise and could never be verified. To my earlier point as well, the difference could just be too small to detect so "accepting the null" and making a precise statement about the difference being 0 is likely to be false anyway.

It may benefit to think of NHST as putting yourself in a dilemma of sorts, as I explain here.

Improve this answer
$\endgroup$
1
  • $\begingroup$ For this OP I recommend we emphasize that "consistent with" does not mean the formal logic meaning of "does not induce a contradiction with". Any probability distribution with suitable support could have generated the observed data, even if it is rather unlikely for many choices of model. So there is an unhelpful notion of logical consistency that could be interpreted there. Instead, as you know, the data is more likely than a set threshold (i.e. the confidence level) to have generated the test statistic computed from the data. $\endgroup$
    – Galen
    Commented Mar 23, 2022 at 3:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.