# What would be the right statistical test and why am I getting different results using Z-test and confidence intervals?

I am trying to determine whether the observed differences in the proportions for two samples are significantly different.

• 0-hypothesis: The proportions are the same for both samples
• Alternative hypothesis: The proportions are different.

These are the known factors:

• SampleSize1 = 200
• SampleSize2 = 1800
• SampleOccurrences1 = 90
• SampleOccurrences2 = 680
• SampleProportion1 = 45 %
• SampleProportion2 = 38 %

I conducted two different tests, but arrive at different conclusions.

Calculating confidence intervals (95 %)

I calculate the confidence interval for each of the samples using Excel:

z = 1.96 StdError = SQRT(SampleProportionX*(1-SampleProportionX)/SampleSizeX) MarginOfError = z * StdError Confidence interval = SampleProportionX +/- MarginOfError

As the confidence intervals are overlapping, I can't reject the 0-hypothesis.

Calculating the Z-value

I calculate the Z-value as follows:

p1 = SampleProportion1 p2 = SampleProportion2 p = (SampleOccurrences1 + SampleOccurrences2) / (SampleSize1 + SampleSize2)

Z = (SampleProportion1 - SampleProportion2) / SQRT(p*(1-p)*(1 / SampleSize1 + 1 / SampleSize2))

As Z > 1.96, I reject the 0-hypothesis.

Why do I get different results, and which test is the correct to use?

You are trying to know whether proportions are the same for both populations. In more formal sense, the hypothesis being tested is:

$$H_0: (p_1 - p_2) =0 \\ H_1: (p_1 -p_2) \neq 0$$

Your second approach (Z-test) is exactly doing this by using Z-statistic. Your first one, not so much.

Your first approach (confidence intervals) is incorrect maybe because it was not clear about "The interval is about what? What value is this confidence interval describing?"

In testing difference between proportion, the value being tested/described is difference. Your first attempt is rather just finding confidence intervals for each population proportion, then examine if the two confidence intervals overlap. Which is not test of difference.

• Checking if confidence overlap can detect differences in proportions, just with markedly less power than a confidence interval for the difference.
– Dave
Aug 5 '20 at 3:50