2
$\begingroup$

I have been using this equation for the two proportions z-test.

enter image description here

However, what if each population sample from which proportions were derived has a different or considerable degree of sample standard error? Is it possible to take sample proportion standard error into account with an equation like this (or perhaps a different one) when comparing two proportions?

Clarifying notes:

Samples are not independent in that month 1 probably has many of the same respondents as month 2.

n1 might be parents of 300 6th grade students for example, where N1 could be parents of all 1000 6th grade students (1 most recent response is accepted per student). n2 and N2 are from the next month of the survey.

From there, I assign a 1 for "very effective" or "effective" and 0 for "Somewhat effective" or "Not effective" then sum 1s and divide by count of all 1s and 0s.

I then compare proportions from month 1 to month 2 using the formula above.

Also of note:

I get the standard error for each month using the following:

Finite Population Correction (FPC): sqrt((N-n)/(N-1))

sample proportion standard error: sqrt(p_success * (1-p_success)/n) * FPC

With that, I want to know if it's possible to somehow take the standard errors (from each month's proportion) into account when conducting the two proportions z-test.

$\endgroup$
5
  • $\begingroup$ Do you mean standard deviation? Despite the similar names, standard deviation and standard error have different meanings. $\endgroup$
    – Dave
    Apr 29 '21 at 22:01
  • $\begingroup$ This is a formula for proportions, as you note; and the quantities with "hats" in the formula refer to fractions of counts. Could you explain to us what you mean by a "standard error" of a count? There are many ways this can come about when a count is not directly measured but is somehow estimated through other means; and you are correct to be concerned that this formula is inappropriate in many such circumstances; but it would help us to know more specifically what your situation is. $\endgroup$
    – whuber
    Apr 29 '21 at 22:34
  • $\begingroup$ Thank you for trying to understand my original post, which I now see was too vague. I have added information to the post which hopefully clarifies what I am trying to do. $\endgroup$ Apr 29 '21 at 23:36
  • $\begingroup$ The finite population correction (often for hypergeometric, rather than binomial models) is often ignored if $n/N < 0.1$ For example if $N = 20,000, n = 1900,$ then it's about $0.95.$ // Now, can you explain how that connects with the z test for equal binomial proportions? Are you sampling without replacement from urns with only a few chips? Or dealing 13-card hands from a 52-card deck? Or trying to predict a mayoral election in a town of population 20,000? $\endgroup$
    – BruceET
    Apr 30 '21 at 0:22
  • $\begingroup$ n is often around 30% of N in my data. I'm polling parents in a school district. $\endgroup$ Apr 30 '21 at 13:35
1
$\begingroup$

There are two widely-used versions of this test:

(a) One takes $H_0$ seriously and approximates the null distribution of the test statistic $Z$ using $\hat p = (x_1 + x_2)/(n_1 + n_2),$ where sample $i = 1,2$ has $x_i$ successes in $n_i$ trials.

(b) The other uses two estimates $\hat p_i = x_i/n_i, i = 1,2.$ The respective standard errors of $\hat p_i$ are estimated by $\sqrt{\frac{\hat p_i(1-\hat p_i)}{n_i}}.$ Then the estimated standard error of $\hat p_1-\hat p_2$ is $$\hat\sigma_{\hat p_1 + \hat p_2}=\sqrt{\frac{\hat p_1(1-\hat p_1)}{n_1} + \frac{\hat p_2(1-\hat p_2)}{n_2}}.$$ Because the CI is not governed by $H_0,$ it make sense to use this version of the estimated standard error for a 95% confidence interval of $p_1-p_2:$ $$\hat p_1 - \hat p_2 \pm 1.96\, \hat\sigma_{\hat p_1 + \hat p_2}.$$

Some statistical software programs use (a) for the test of $H_0: p_1 = p_2$ against $H_a: p_1 \ne p_2,$ using your formula for the test statistic $Z,$ but showing a CI that uses method (b). Others use method (b) throughout with the 'separate variance' version of the estimated standard error in the denominator of $Z$ and also in the CI.

I believe that prop.test in R uses (b) throughout. Minitab gives one the choice between the 'pooled' estimate of SE in (a) for the test or the 'separate variances' estimate of SE in (b).

For example, consider $x_1 = 135, n_1 = 500; x_2 = 130, n_2 = 600.$ then in R we have:

x1 = 135;  n1 = 500
x2 = 130;  n2 = 600
prop.test(c(x1,x2), c(n1,n2), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(x1, x2) out of c(n1, n2)
X-squared = 4.2421, df = 1, p-value = 0.03943
alternative hypothesis: two.sided
95 percent confidence interval:
 0.002333942 0.104332725
sample estimates:
   prop 1    prop 2 
0.2700000 0.2166667 

Mintab: Pooled

Test and CI for Two Proportions 

Sample    X    N  Sample p
1       135  500  0.270000
2       130  600  0.216667

Difference = p (1) - p (2)
Estimate for difference:  0.0533333
95% CI for difference:  (0.00233394, 0.104333)
Test for difference = 0 (vs ≠ 0):  
   Z = 2.06  P-Value = 0.039

Minitab: Separate variances.

Test and CI for Two Proportions 

Sample    X    N  Sample p
1       135  500  0.270000
2       130  600  0.216667

Difference = p (1) - p (2)
Estimate for difference:  0.0533333
95% CI for difference:  (0.00233394, 0.104333)
Test for difference = 0 (vs ≠ 0):  
  Z = 2.05  P-Value = 0.040

Notice that there is a slight difference between the test statistics and the corresponding P-values in the two Minitab versions. But both versions show the CI from the 'separate variances' method (b).

Both Minitab versions also report the P-value 0.04 from Fisher's Exact test, which is an exact hypergeometric probability, not a normal approximation. In R, Fisher's exact test uses the $2 \times 2$ contingency table TBL. Results agree with those in Minitab.

x = c(135, 130);  n = c(500, 600)
TBL = rbind(x, n-x);  TBL
  [,1] [,2]
x  135  130
   365  470

fisher.test(TBL)

        Fisher's Exact Test for Count Data

data:  TBL
p-value = 0.04036
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 1.003776 1.781126
sample estimates:
odds ratio 
  1.336824 

[Notice that the CI in the output of fisher.test is for an odds ratio.]

$\endgroup$
3
  • $\begingroup$ I believe none of this answer responds to the situation described in the question: see my comment above. $\endgroup$
    – whuber
    Apr 29 '21 at 22:34
  • $\begingroup$ You may be right; the question is vague (maybe even notationally wrong). But I doubt the question came out of nowhere, so I tried to guess what might be behind it. $\endgroup$
    – BruceET
    Apr 29 '21 at 22:40
  • 1
    $\begingroup$ When you find yourself guessing, it's always better to ask for clarification. $\endgroup$
    – whuber
    Apr 29 '21 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.