# Comparing proportions in two samples

(x-posted to Statalist)

I have two (probably) independent samples taken from the same population under two different surveys conducted a few years apart. I'd like to check whether the frequencies for a particular indicator match between the samples. I don't expect these proportions to change much over time (though I can't rule it out). The indicator uses multiple categories which are mutually exclusive and not ordinal, e.g.:

1. Red
2. Green
3. Black
4. Blue

etc.

I want to check that survey A captured, basically, the same proportions of red/green/blue that survey B did. I think I should be using a Chi-square goodness of fit for this (csgof, in Stata); except that I'm comparing two samples, rather than the sample versus the population. Anyone know what other statistical test I should be using? I'm a bit stuck, and am considering just eyeballing it: using svy: tab (in Stata) on both and just comparing the confidence intervals. But I feel like this is a crude way of doing things.

Thanks very much!

a

• That's a test of homogeneity (of proportions). The usual test follows the same calculation as a chi-square test of independence. – Glen_b -Reinstate Monica Jun 4 '14 at 23:13
• Why does this include the tags t-test and z-test? They don't seem to be at all relevant. Strictly, goodness-of-fit doesn't really apply here either. – Glen_b -Reinstate Monica Jun 4 '14 at 23:45
• @Glen_b: worth updating my answer to test of homogeneity instead of independence? Or is there really no difference? These chi squares always throw me for a loop... – Nick Stauner Jun 5 '14 at 16:28
• @Nick I thought your answer was fine as it stood. As far as I know, the only potential difference occurs in the situation where you don't automatically condition on both margins. [Nearly always, I tend to condition on both margins ... but I don't want to enter into the merits of the decades-long arguments there, fascinating as it is. When I am being a Bayesian at least, there's a difference - homogeneity would be 'condition on one margin', while independence would be 'condition on both'.) ... (ctd) – Glen_b -Reinstate Monica Jun 5 '14 at 22:57
• @Nick (ctd) ... Indeed, many people define the two by the conditioning you do (which since you condition on both means your discussion is 100% correct); this is consistent, since when you condition on both they are identical - homgeneity is independence. If you are concerned, it might be worth linking to some of the discussions of the difference, such as this, but I wouldn't worry - your 'You could use' makes your answer perfectly correct. – Glen_b -Reinstate Monica Jun 5 '14 at 23:02

You could use a $\chi^2$ test of independence to compare frequencies from surveys A and B. As usual, $$\chi^2=\Sigma\frac{(O-E)^2}{E}$$But your expected values will differ for each cell of the table:$$E_{(r,c)} = \frac{n_r n_c}N\\\begin{array}{c|cccc|c}&\rm Red&\rm Green&\rm Black&\rm Blue&\rm Total\\\hline\text{Survey A}&n_{(1,1)}&n_{(1,2)}&n_{(1,3)}&n_{(1,4)}&n_{(1,\rm all)}\\\text{Survey B}&n_{(2,1)}&n_{(2,2)}&n_{(2,3)}&n_{(2,4)}&n_{(2,\rm all)}\\\hline\rm Total&n_{(\rm all,1)}&n_{(\rm all,2)}&n_{(\rm all,3)}&n_{(\rm all,4)}&N\end{array}$$
What effect size estimate? you might ask. (Briefly, since you didn't...) Cramér's $\phi$. For more on that, see "Chi squared test with expected frequencies coming from another observation".