3
$\begingroup$

Basic stats question that's been bothering me. Let's say I've put out a survey question with two possible answers, and have gotten back responses that have overlapping confidence intervals (at 95%).

Two bars with overlapping confidence interval lines.

This means I can't yet say one answer is statistically more likely than the other.

But how do these results differ from something like this:

enter image description here

In this example, there's far less overlap between the two confidence intervals. Is there some mathematical way that represents a higher likelihood of a significant result, or is this wishful thinking?

Improve this question
$\endgroup$
3
  • $\begingroup$ It is possible that the confidence intervals overlap, yet are significantly different. See articles here and here. $\endgroup$
    – william3031
    Commented Aug 25, 2019 at 11:57
  • 4
    $\begingroup$ By making mild assumptions, you actually can draw conclusions based on examining overlap or non-overlap of confidence intervals: see stats.stackexchange.com/questions/18215. You cannot, however, draw inferences about "statistical likelihood" without making very strong assumptions about prior probabilities. $\endgroup$
    – whuber ♦
    Commented Aug 25, 2019 at 12:47
  • $\begingroup$ I think there's confusion over estimation versus testing here. Also, the participants are asked to choose between 2 options? If so, there is a constraint a + b = 1, which makes the plots quite misleading in my opinion $\endgroup$
    – innisfree
    Commented Dec 16, 2020 at 9:25

2 Answers 2

Reset to default
3
$\begingroup$

A useful, but approximate, rule of thumb is that when the overlap in 95% CIs is less than half of the margin of error, the difference will be significant at $p \le 0.05$. Margin of error is a fancy way of saying half the length of the confidence interval, or one "whisker."

When the 95% CIs are just touching or completely disjoint, the difference will be highly significant, with $p \le 0.01$.

In other words, you need a fair amount of overlap before the data cannot distinguish between the two groups.

This ROT is sufficiently accurate when both sample sizes are at least 10 and margins of error do not differ by more than a factor of two, even when the sample sizes are not equal.

Improve this answer
$\endgroup$
0
$\begingroup$

If you have accepted the criterion that you will consider treatments different if their confidence intervals don't overlap, then that's your decision criterion... That being said, it doesn't make a lot of sense in reality to have a sharp cutoff for these decisions... To describe the difference between treatments, you might look at some kind of effect size statistic, like the difference between means, or Cohen's d.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.