2 Treatments, comparison of proportions. Z test?

Question

My design is the following: two treatments, T1 and T2. Population is divided in 4 age brackets, and the absolute numbers of enrolled subjects are:

Age	Treat1	Treat2
12-39	1387907	12196048
40-59	876762	14314721
60-79	295045	11928111
80plus	93709	4233887

The outcome. i.e. subjects showing a certain effect, are:

Age	Treat1	Treat2
12-39	31	90
40-59	32	219
60-79	49	737
80plus	33	991

Now, I'd like to see if there is a significant difference between Treat1 and Treat2, given this outcome. My idea was to perform a 2 sample z-test for proportions for each age group. E.g., for 12-39:

p1 = 31/1387907 = 0.000022
N1 = 1387907

p2 = 90/12196048 = 0.000007
N2 = 12196048

Result of two tailed z-test for two proportions:
z = 5.7
p-val < 0.0001
So this looks significant.

By repeating this for all the age groups, I get that they are all significantly different. Now, is this the correct way to test this data? If so, next question: why is it that, when I pool all the subjects together (disregarding the age groups) and perform a z-test, it comes out non-significant? E.g.

p1 = (31+32+49+33)/(1387907+876762+295045+93709) = 0.000055
N1 = 2653423

p2 = (90+219+737+991)/(12196048+14314721+11928111+4233887) = 0.000048
N2 = 42672767

Result of two-tailed z-test for two proportions:
z = 1.6
p-val = 0.1118

Is this normal, even if it looks a little counterintuitive to me?

Thanks for helping a total rookie :)

Answer 1

You are dealing with very small proportions, so detecting an effect requires relevant sample size. For the last age group (80+), the test significant is if $\alpha = 0.05$ but not if $\alpha = 0.01$. prop.test returns a p-value of 0.02.

If we run power.prop.test() to determine the sample size required for your last test (total) (indicative power of 0.8):

power.prop.test(p1=0.000055, p2=0.000048, power=0.8, alternative="two.sided")

The command returns that $n$ should be equal to 16,497,815 in each group which is not the case for Treat1.

For the sake of the exercise and to illustrate the impact of the sample size on your test, if you multiply both $x_1$ and $n_1$ (Treat1) by 10 in the last age group 80+ (and therefore holding the same proportion), the result of the test for all age groups (total) will become significant.

So, you are doing the right test and it returns the correct result. Your experiment returns low proportions so you might want to keep an eye on power.