10
$\begingroup$

It is obviously fallacious to assume that failure to reject the null implies that the null is true. But in a case where the null is not rejected and the corresponding confidence interval (CI) is narrow and centered around 0, does this not provide evidence for the null?

I am of two minds: Yes, in practice this would provide evidence that the effect is more or less 0. However, in a strict hypothesis-testing framework, it seems that null effects are simply unusable for inference, as are their corresponding CIs. So what is the meaning of a CI when its point estimate is non-significant? Is it also unusable for inference or can it be used as in the preceding example to quantify evidence for the null?

Answers with scholarly references are encouraged.

Improve this question
$\endgroup$
3
  • $\begingroup$ You will probably be interested in equivalence testing and questions on the site detailing it. See How to test hypothesis of no group differences? for one example. $\endgroup$
    – Andy W
    Apr 6, 2014 at 18:07
  • 1
    $\begingroup$ If you mean evidence for a point null against the alternative of anything else ... then, no. The uncountably infinite number of alternatives between the observed very small value and the null are still going to be more likely than the null. If you mean something else, then perhaps in some circumstances. $\endgroup$
    – Glen_b
    Apr 6, 2014 at 19:48
  • $\begingroup$ Yes, then it would be a matter of equivalent testing, a term I'd not yet heard of. $\endgroup$
    – ATJ
    Apr 6, 2014 at 20:35

2 Answers 2

Reset to default
7
$\begingroup$

In short: Yes.

As Andy W wrote, concluding that the parameter equals a specified value (in your case, effect size equals zero), is a matter of equivalence testing.

In your case, this narrow confidence interval may in fact indicate that the effect is practically zero, that means, the equivalence's null hypothesis may be rejected. Significant equivalence at $1-\alpha$-level is typically shown by an ordinary $1-2\alpha$-confidence interval that lies completely within a prespecified equivalence interval. This equivalence interval takes into account that you are able to neglect really tiny deviations, i.e. all effect sizes within this equivalence interval can be considered to be practically equivalent. (Statistical test of equality are not possible.)

Please see Stefan Wellek's "Testing Statistical Hypotheses of Equivalence and Noninferiority" for further reading, the most comprehensive book on this matter.

Improve this answer
$\endgroup$
2
$\begingroup$

Null hypotheses exemplify the meaning of "All models are wrong, but some are useful." They're probably most useful if not taken literally and out of context – that is, it's important to remember the epistemic purpose of the null. If it can be falsified, which is the intended objective, then the alternative becomes more useful by comparison, albeit still rather uninformative. If you reject the null, you're saying the effect is probably not zero (or whatever – null hypotheses can specify other values for falsification too)...so what is it then?

The effect size you calculate is your best point estimate of the population parameter. Generally, chances should be equally good that it's an overestimate or underestimate, but the chances that it's a dead-center bulls-eye are infinitesimal, as @Glen_b's comment implies. If by some bizarre twist of fate (or by construction – either way, I assume we're speaking hypothetically?) your estimate falls directly on $0.\bar 0$, this is still not much evidence that the parameter is not a different value within the confidence interval. The meaning of the confidence interval does not change based on the significance of any hypothesis test, except in as much as it may change location and width in a related way.

In case you're not familiar with what effect size estimates look like for samples from a (simulated) population of which the null hypothesis is literally true (or in case you haven't seen it yet and are just here for a little statistical entertainment), check out Geoff Cumming's Dance of the $p$ Values. In case those confidence intervals aren't narrow enough for your taste, I've tried simulating some of my own in R using randomly generated samples just shy of $n=1\rm M$ each from $\mathcal N(0,1)$. I forgot to set a seed, but set x=c() and then ran x=append(x,replicate(500,cor(rnorm(999999),rnorm(999999)))) as many times as I cared to before finishing this answer, which gave me 6000 samples in the end. Here's a histogram and a density plot using hist(x,n=length(x)/100) and plot(density(x)), respectively:

$\ \ \ \ $

As one would expect, there's evidence for a variety of nonzero effects arising from these random samples of a population with literally zero effect, and these estimates are distributed more or less normally around the true parameter (skew(x)= -.005, kurtosis(x)= 2.85). Imagine you only knew the value of your estimate from a sample of $n=1\rm M$, not the true parameter: why would you expect the parameter to be closer to zero than your estimate instead of further? Your confidence interval might include the null, but the null isn't really any more plausible than the value of equivalent distance from your sample effect size in the opposite direction, and other values may be more plausible than that, especially your point estimate!

If, in practice, you want to demonstrate that an effect is more or less zero, you need to define how much more or less you're inclined to ignore. With these huge samples I've simulated, the estimate of largest magnitude I generated was $|r|=.004$. With more realistic samples of $n=999$, the largest I find among $1\rm M$ samples is $|r|=.14$. Again, the residuals are normally distributed, so these are unlikely, but the point is they're not implausible.

A CI is probably more useful for inference than a NHST in general. It doesn't just represent how bad an idea it might be to assume the parameter is negligibly small; it represents a good idea of what the parameter actually is. One can still decide whether this is negligible, but can also get a sense of how non-negligible it could be. For further advocacy of confidence intervals, see Cumming (2014, 2013).

References
- Cumming, G. (2013). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge.
- Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(7), 7–29. Retrieved from http://pss.sagepub.com/content/25/1/7.full.pdf+html.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks, I'm highly familiar with Cumming's work. I suppose my question was more along the lines of, "if the point ES estimate is nonsignificant, then can the CIs be used for inference? (Or are they 'null' i.e., useless as the point estimate)" $\endgroup$
    – ATJ
    Apr 7, 2014 at 11:33
  • 1
    $\begingroup$ @ATJ: Neither the point estimate of nor the ($1-\alpha$) confidence intervals for a parameter become "useless" when not significantly different from zero (at level $\alpha$) or containing zero respectively. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Apr 7, 2014 at 12:42
  • $\begingroup$ @ATJ: As I said, the meaning[/utility] of a CI doesn't change based on the significance of any NHST. A CI is probably more useful for inference than a NHST in general...it represents a good idea of what the parameter actually is. E.g., I just ran cor.test(rnorm(9999999),rnorm(9999999)) and got a CI of $\{-0.00063,0.00060\}$. Therefore I infer that when I run it again, I am 95% likely to get a new estimate within that range. Running it again, my estimate was $r=0.00029$; my CI-based inference was right! The null happens to be by construction, but my evidence would favor my estimate instead... $\endgroup$
    – Nick Stauner
    Apr 7, 2014 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.