3
$\begingroup$

I have calculated a z test to examine the difference between two proportions. I found differences to be significant. The formula I used was:

$$z = (p_1-p_2)/SE$$ where $$SE = \sqrt{ p ( 1 - p ) ( \frac{1}{n_1} + \frac{1}{n_2} ) }$$

and $n_1$ is the sample size of sample 1, $n_2$ sample size of sample 2.

I have also plotted a bar graph of my proportions with 95% confidence intervals around each of the two proportions based on this equation:

For proportion 1:
$p_1 \pm 1.96SE$

$SE = \sqrt{p(p-1)/ n_1}$; $n_1$: size of the sample 1

For proportion 2:

$p_2 \pm 1.96SE$

$SE = \sqrt{p(p-1)/ n_2}$; $n_2$: size of the sample 2

I would not expect the CIs to overlap much based on significant of z test. However, they overlap by more than 25%. Am I using the correct formula for the confidence intervals? If so why does there seem to be discrepancy between the results of the z test and the visual depiction of the difference

Improve this question
$\endgroup$
2
  • $\begingroup$ Plot the CI for the quantity you're interested in - the difference of the proportions. $\endgroup$
    – jona
    May 24, 2014 at 17:43
  • 1
    $\begingroup$ A quantitative analysis of this situation is provided at stats.stackexchange.com/questions/18215. Although it's probably TMI, it definitely answers your question! $\endgroup$
    – whuber ♦
    May 24, 2014 at 19:04

2 Answers 2

Reset to default
4
$\begingroup$

Statistics is a pretty large field and it is good to invest in understanding the fundamentals before attempting to use statistics. What you stated is a common misconception about statistics. Individual confidence intervals make an inference about one quantity of interest. You need a confidence interval for the difference in two such quantities. There are many confidence intervals developed for the difference in two probabilities. It is easy to have individual intervals overlap whereas the interval for the difference excludes zero. The converse is not true however. If the interval for the difference includes zero, the two individual intervals must overlap. Note that it is not typical for the two individual intervals to be of much interest in study in which subjects are not a simple random sample from the population to which inference is desired.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi. Thanks for your answer, it is helpful. I have one more question. I will plot the CI for the difference, but I would also like to plot the individual proportions next to each other on a bar graph. People usually expect to see the 95% CI on each bar. Is it meaningless to plot this? If I do plot the individual CIs for each bar is there a limit for the expected to overlap (e.g. by 25%). My CIs overlap by approx 40% but the z test still shows significance. $\endgroup$
    – user45114
    May 25, 2014 at 11:33
  • $\begingroup$ It is unclear why you are still concerned about the relationship between the two different types of confidence intervals. But the answer below gives you the details. Don't use bar charts as they give a distorted impression about the intervals. Whether it's meaningless depends entirely on your sampling strategy. $\endgroup$
    – Frank Harrell
    May 25, 2014 at 19:55
  • $\begingroup$ my sampling statergy was a random sample from the population of interest. I would like to plot a bar graph of two proportions next to each other. People usually expect to see an confidence interval on each of the bars. I'm simply asking whether it is meaningless to plot confidence intervals for each bar, using the formula above (for the CI for each group) - I sense that the answer is yes. In which case why do people do this (all the time!). Why does using bar chart distort the difference between the two proportions? $\endgroup$
    – user45114
    May 26, 2014 at 11:56
  • $\begingroup$ It is not meaningless but from the standpoint of helpfulness I'd put it at 0.4. I think that it's 0.1 if the samples are not random samples from the population of interest. Summed up, as Bill Cleveland has said, if you want to learn about a difference, show the difference. $\endgroup$
    – Frank Harrell
    May 26, 2014 at 13:18
1
$\begingroup$

As you can see, the standard error for the test or, equivalently, for CI of the difference, involved the term $(\frac{1}{n_1}+\frac{1}{n_2})$ where the single confidence intervals only had $\frac{1}{n_i}$. Clearly, the former is larger making the confidence interval for the difference smaller.

The other point making the CI for the difference smaller, is that the degree of freedom also becomes smaller. This in turn makes the 95%-quantiles smaller. (However, this consideration only applies to non-asymptotic tests/CIs.)

That's why overlapping confidence intervals do not imply the confidence interval of the difference to cover the $0$, whereas covering $0$ implies overlap.

Improve this answer
$\endgroup$
2
  • $\begingroup$ if you edit again, you might want to change 'intercal' to 'interval'. I couldn't figure out what you meant for a minute. $\endgroup$
    – Glen_b
    May 25, 2014 at 1:02
  • $\begingroup$ Thank you, I thought that it could be something to do with 'more power' for the z test because it was using N of both samples. Is this correct? Can i adjust my individual CIs to account for this? So that they are more representative of the CI of the difference? $\endgroup$
    – user45114
    May 25, 2014 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.