3
$\begingroup$

I have many pairs of measurements. A paired-samples t-test on these produces a 2-tailed p > .1 (so non-significant by typical alpha values).

When I run two separate one-sample t-tests comparing against some common null (say 0), one such test produces a p ~= .01 (therefore significant), while the other produces p ~= .6 (therefore non-significant).

The 95% confidence intervals of the two measurements overlap substantially. The non-significant measurement's 95CI additionally overlaps 0, whereas the significant measurement's 95CI does not overlap 0.

So whereas the pairwise differences between the measurements are too small to reach significance, one is significantly different from 0 overall while the other is not.

There seems to be a logical contradiction between saying that the two measurements do not differ, when they differ in their proximity to a common value.

Improve this question
$\endgroup$
1
  • $\begingroup$ I am also interested in whether one or other of the approaches in my question is [in]valid for the problem at hand, and if they are both valid (while permitting different interpretations) what difference this makes for 'decision behaviour'. I often see the 'double t-test' approach deployed when the pairwise test fails to reach significance. People talk up their understanding of the conceptual difference yet go on to treat the latter result essentially as if the former (pairwise result) had worked. Hope that makes sense $\endgroup$
    – benxyzzy
    Commented Jul 27, 2017 at 12:25

1 Answer 1

Reset to default
5
$\begingroup$

There is no contradiction, because The Difference Between "Significant" and "Not Significant" is not Itself Statistically Significant (Gelman & Stern, The American Statistician, 2006). Gelman posted a PDF version of the article here.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ +1. See also nature.com/neuro/journal/v14/n9/abs/nn.2886.html. $\endgroup$
    – amoeba
    Commented Jul 27, 2017 at 12:12
  • $\begingroup$ Thank you both for the links, they look insightful. I may post a follow-up question asking whether any test exists which jointly compares two groups and a common reference value, so as to avoid computing two separate tests against the value and then erroneously comparing the tests' significances. $\endgroup$
    – benxyzzy
    Commented Jul 27, 2017 at 12:16
  • $\begingroup$ I'm looking forward to that follow-up question (best to link the two questions together explicitly). You'll need to be specific as to which alternative hypothesis you'd be testing: that either or both groups differ from the reference. $\endgroup$
    – Stephan Kolassa
    Commented Jul 27, 2017 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.