Comparing proportions in two samples

Question

(x-posted to Statalist)

I have two (probably) independent samples taken from the same population under two different surveys conducted a few years apart. I'd like to check whether the frequencies for a particular indicator match between the samples. I don't expect these proportions to change much over time (though I can't rule it out). The indicator uses multiple categories which are mutually exclusive and not ordinal, e.g.:

What is your car's color?

1. Red
2. Green
3. Black
4. Blue

etc.

I want to check that survey A captured, basically, the same proportions of red/green/blue that survey B did. I think I should be using a Chi-square goodness of fit for this (csgof, in Stata); except that I'm comparing two samples, rather than the sample versus the population. Anyone know what other statistical test I should be using? I'm a bit stuck, and am considering just eyeballing it: using svy: tab (in Stata) on both and just comparing the confidence intervals. But I feel like this is a crude way of doing things.

Thanks very much!

a

Answer 1

^{This answer borrows material from another of mine on "Formula to compare response rates". See that one as well!}
You could use a $\chi^2$ test of independence to compare frequencies from surveys A and B. As usual, $$\chi^2=\Sigma\frac{(O-E)^2}{E}$$But your expected values will differ for each cell of the table:$$E_{(r,c)} = \frac{n_r n_c}N\\\begin{array}{c|cccc|c}&\rm Red&\rm Green&\rm Black&\rm Blue&\rm Total\\\hline\text{Survey A}&n_{(1,1)}&n_{(1,2)}&n_{(1,3)}&n_{(1,4)}&n_{(1,\rm all)}\\\text{Survey B}&n_{(2,1)}&n_{(2,2)}&n_{(2,3)}&n_{(2,4)}&n_{(2,\rm all)}\\\hline\rm Total&n_{(\rm all,1)}&n_{(\rm all,2)}&n_{(\rm all,3)}&n_{(\rm all,4)}&N\end{array}$$

In your case, your null hypothesis is that the proportions of each color will be similar for both surveys. Most hypothesis testers like to set a rule before performing their significance tests that they'll reject the null hypothesis model if there's less than a 5% chance of getting data that violates the model's expectations as much as or more than their sample data does if the model is true of the population from which the sample was randomly selected. It's not really necessary (and may even be inadvisable) to just dichotomize your attitude toward the null hypothesis model into a decision of whether to reject it wholesale, but that's the conventional approach, so it's probably what others would expect you to do.

It's worth noting that there's a Bayesian alternative, whereas the significance testing approach I've just described follows frequentist theory. One may also choose to be more interested in estimating the effect size and the level of confidence with which one can place that estimate within a confidence interval (or margin of error ×2) first and foremost, and consider the question of whether the margin of error on the side of zero includes the null hypothesis as a secondary concern. It's definitely not "crude" to prefer this approach. If anything, it's more crude to ignore your actual effect size estimate just because you're less than 95% confident that it would be at least as large if you repeated the study and the null is literally true (which it often isn't).

What effect size estimate? you might ask. (Briefly, since you didn't...) Cramér's $\phi$. For more on that, see "Chi squared test with expected frequencies coming from another observation".