5
$\begingroup$

My question is relatively simple, and has to do with reporting seemingly contradictory results from statistical tests. This is the problem.

There are two independent groups, and two research questions are important:

  1. Are the two (population) group means equal
  2. Do both (population) means differ significantly from zero?

An independent groups t-test leads to the conclusion (A) that both group means do NOT differ significantly from one another. Further, two “one sample t-tests”, lead to the conclusion (B) that group-mean 1 differs significantly from zero, whereas group-mean 2 does NOT.

These two test conclusions seem contradictory, because for the true means in the populations conclusion (A) and (B) could not both be valid. Sample tests, of course, can lead to such contradictory conclusions. Hence my question is: how to deal with such results, or how to report them? Simply reporting both conclusions (A) and (B) is not "wrong" but it feels inconsistent. Any thoughts about this?

Improve this question
$\endgroup$
1
  • $\begingroup$ Your first question means you want to use an equivalence test (e.g., using tost) of a null hypothesis like $\text{H}_{0}\text{: }|\mu_{1} - \mu_{2}| \ge \Delta$, with $\text{H}_{\text{A}}\text{: }|\mu_1 - \mu_1| < \Delta$, where $\Delta$ is the boundary between "large enough of a difference in means to care about" and "small enough a difference in means, that we consider the means equivalent." $\endgroup$
    – Alexis
    Commented Dec 25, 2021 at 21:01

3 Answers 3

Reset to default
8
$\begingroup$

This is indeed a problem (one of many) with hypothesis testing. This particular scenario is discussed in detail in the paper by Andrew Gelman and Hal Stern titled The Difference Between "Significant" and "Not Significant" is not Itself Statistically Significant

From the abstract:

The ubiquity of this statistical error leads us to suggest that students and practitioners be made more aware that the difference between “significant” and “not significant” is not itself statistically significant.

From the discussion:

Statistical significance, in some form, is a way to assess the reliability of statistical findings. However, as we have seen, comparisons of the sort, “X is statistically significant but Y is not,” can be misleading.

Reference

Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328-331.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for this reference, I will certainly read that! $\endgroup$
    – BenP
    Commented Dec 7, 2021 at 15:32
4
$\begingroup$

Consider the simplified case of intersecting intervals (see below). It can be the case to have three intervals such that A and C both overlap with B, but A and C don't overlap with each other.

enter image description here

To translate this to hypothesis testing, think of the test as of looking if confidence intervals overlap.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ +1, I think this visualization is helpful, but it doesn't really address the apparent contradiction. This does show that this problem isn't limited to hypothesis testing, as the problem doesn't go away if we use confidence intervals. This is also discussed in the Gelman & Stern paper (linked in another answer). $\endgroup$
    – knrumsey
    Commented Dec 7, 2021 at 17:50
  • $\begingroup$ Thanks for the CI drawing, Tom, it indeed helps to show the problem, like knrumsey already noticed. The paper of Gelman and Stern reminds me of what is common practice in anova settings: only proceed to multiple comparisons of group means, if the global F test is significant. And even then, be cautious not to have a too high family-wise error rate, by applying a Bonferroni or whatever method is. Although in my example and Gelman's 2-groups example, the comparisons are not made among group means, but between group means (or b-coefficients) and some external and interesting value, like 0. $\endgroup$
    – BenP
    Commented Dec 10, 2021 at 10:59
  • $\begingroup$ @BenPelzer comparing intervals and comparing the means to zero is the same thing. In first case you compare if the intervals overlap. In the second case, if the interval over the difference between the means overlaps with zero. See stats.stackexchange.com/questions/18215/… $\endgroup$
    – Tim
    Commented Dec 10, 2021 at 11:21
  • $\begingroup$ @knrumsey there is no apparent contradiction. These are different problems. Whether or not two means are the same has no implications for whether or not either or both individual means equal zero. See my answer below. $\endgroup$
    – Graham Bornholt
    Commented Dec 27, 2021 at 22:55
  • $\begingroup$ @GrahamBornholt The contradiction arises in the incorrect but common interpretation of hypothesis testing results. "We conclude that $A = 0$ and $B = 0$ but $A \neq B$". Of course this interpretation is incorrect, but it is an easy trap to fall into and thus worth discussion (see the paper linked in another answer for such a discussion). $\endgroup$
    – knrumsey
    Commented Jan 4, 2022 at 15:59
1
$\begingroup$

It is a completely acceptable situation that can happen easily. Informally, (and ignoring volatility differences), it just means that both group sample means are not that far apart, but the group 2 sample mean is a bit closer to 0 than is the group 1 sample mean so they fall on different sides of their respective p-value thresholds. Say, the group 2 mean is 1.8 and the group 1 mean is 2.2. The sample difference is just 0.4, much closer to 0 than either of its component means. That's just the nature of testing and sample variability.

There is an important fact about the volatility of differences that impacts on this. The variance of the difference of two independent sample means is the sum of the individual variances. So a difference is much more volatile than its components, requiring bigger differences to achieve low p-values!

Improve this answer
$\endgroup$
1
  • $\begingroup$ The paper linked in the accepted answer describes a similar but more extreme example, where we conclude that $A$ does not differ from $0$ and $B$ does not differ from $0$ but $A$ and $B$ differ from each other. I don't think that your answer gets to the root of the issue, as it relies on the statement "a difference is much more volatile than it's components". $\endgroup$
    – knrumsey
    Commented Jan 4, 2022 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.