1
$\begingroup$

According to to G. Cumming and S. Finch in their paper "Inference by Eye" the 95% confidence intervals of two independent means can overlap and the difference between the two means can still be significantly different from 0. They found that at approximately 58% overlap the p-value of the t-test changes from significant to not significant at 0.05.

Now to my question.

Does this apply for all types of confidence intervals? Im mostly interested if this holds for two independent proportions. If yes, why is that? And if no, why is that?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Is the confidence interval approximated using the normal distribution? $\endgroup$
    – user2974951
    Commented Nov 22, 2023 at 13:08
  • $\begingroup$ For the two proportions? Yes the confidence intervals is approximated using the normal distribution. In my case n1 and n2 are both > 10 000 and p1 and p2 are both around 0,1. $\endgroup$
    – Jam.Wil
    Commented Nov 22, 2023 at 13:14
  • 2
    $\begingroup$ stats.stackexchange.com/questions/18215 answers your question generally and offers a more specific, quantitative solution than the paper you reference. $\endgroup$
    – whuber ♦
    Commented Nov 22, 2023 at 14:48

1 Answer 1

Reset to default
1
$\begingroup$

I hesitate to say it applies to all types of confidence intervals, there could be some odd cases. But it generally applies (although it can get tricky).

Why?

Well, a p-value of x (say, 0.05) would match a 1-x (0.95) confidence interval vs. a constant. For the two means case, it would be that the CI for one proportion doesn't cover the other propotion (not its CI, but the value itself).

It gets tricky for more complex notions. E.g. if we have p means, then which are overlapping which? And what the "other" CI would be is not at all clear for, say, an F test. But the idea is generally sound. It certainly applies to, say, a t-test of means rather than proportions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.