1
$\begingroup$

I want to test for differences between two proportions, specifically, a complete count of the number of events occurring within a population in year 1 vs that in year 2. I thought a two proportion z test would be appropriate. However, the size of the denominator is very large and consistently results in a small test statistic and significant results. Is there an adjustment or method to account for the large population size? Is there a different test that is more appropriate?

P = ((B1p1)+(B2p2))/(B1+B2)
Test Statistic (alpha=0.05) = 1.96 * sqrt [(P*(1-P))*((1/B1)+(1/B2))]

B1 = 2.02billion; p1 = 15.5%
B2 = 2.04billion; p2 = 18.2%
Percent difference = 18.2%-15.5% = 2.7%
P = 0.16857 Test statistic = 2.3E-05
2.7% > 2.3E-05 -> significant

Improve this question
$\endgroup$
1
  • $\begingroup$ Why do you think you are getting a result in need of adjusting? $\endgroup$
    – Dave
    Commented Sep 7, 2021 at 14:17

1 Answer 1

Reset to default
3
$\begingroup$

Is there an adjustment or method to account for the large population size?

This is a feature, not a bug. Immense sample sizes yield enormous precision, and so the change is likely "statistically significant" (because no two years are exactly alike, so the null is a straw man to start with) but may not always be practically significant.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks @Demetri Pananos. I suppose I asked about alternative methods / adjustments out of limited experience with such large N. Is there another test that is more appropriate? Or is testing not appropriate? I am trying to understand the boundaries within which I can convey differences year to year. $\endgroup$
    – sb_2006
    Commented Sep 7, 2021 at 14:32
  • $\begingroup$ In my own opinion, when the sample size is in the Billions, statistical significance is not what you want. You should consider any biases in your data, as those are the larger threat now as opposed to variance. It all depends on what your goal is. If you just want evidence that things changed year over year, you have it and don't need to do a test. $\endgroup$
    – Demetri Pananos
    Commented Sep 7, 2021 at 14:49
  • $\begingroup$ thanks for your help! :) $\endgroup$
    – sb_2006
    Commented Sep 8, 2021 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.