1
$\begingroup$

(x-posted to Statalist)

I have two (probably) independent samples taken from the same population under two different surveys conducted a few years apart. I'd like to check whether the frequencies for a particular indicator match between the samples. I don't expect these proportions to change much over time (though I can't rule it out). The indicator uses multiple categories which are mutually exclusive and not ordinal, e.g.:

What is your car's color?

  1. Red
  2. Green
  3. Black
  4. Blue

etc.

I want to check that survey A captured, basically, the same proportions of red/green/blue that survey B did. I think I should be using a Chi-square goodness of fit for this (csgof, in Stata); except that I'm comparing two samples, rather than the sample versus the population. Anyone know what other statistical test I should be using? I'm a bit stuck, and am considering just eyeballing it: using svy: tab (in Stata) on both and just comparing the confidence intervals. But I feel like this is a crude way of doing things.

Thanks very much!

a

$\endgroup$
  • $\begingroup$ That's a test of homogeneity (of proportions). The usual test follows the same calculation as a chi-square test of independence. $\endgroup$ – Glen_b Jun 4 '14 at 23:13
  • $\begingroup$ Why does this include the tags t-test and z-test? They don't seem to be at all relevant. Strictly, goodness-of-fit doesn't really apply here either. $\endgroup$ – Glen_b Jun 4 '14 at 23:45
  • $\begingroup$ @Glen_b: worth updating my answer to test of homogeneity instead of independence? Or is there really no difference? These chi squares always throw me for a loop... $\endgroup$ – Nick Stauner Jun 5 '14 at 16:28
  • 1
    $\begingroup$ @Nick I thought your answer was fine as it stood. As far as I know, the only potential difference occurs in the situation where you don't automatically condition on both margins. [Nearly always, I tend to condition on both margins ... but I don't want to enter into the merits of the decades-long arguments there, fascinating as it is. When I am being a Bayesian at least, there's a difference - homogeneity would be 'condition on one margin', while independence would be 'condition on both'.) ... (ctd) $\endgroup$ – Glen_b Jun 5 '14 at 22:57
  • 1
    $\begingroup$ @Nick (ctd) ... Indeed, many people define the two by the conditioning you do (which since you condition on both means your discussion is 100% correct); this is consistent, since when you condition on both they are identical - homgeneity is independence. If you are concerned, it might be worth linking to some of the discussions of the difference, such as this, but I wouldn't worry - your 'You could use' makes your answer perfectly correct. $\endgroup$ – Glen_b Jun 5 '14 at 23:02
2
$\begingroup$

This answer borrows material from another of mine on "Formula to compare response rates". See that one as well!
You could use a $\chi^2$ test of independence to compare frequencies from surveys A and B. As usual, $$\chi^2=\Sigma\frac{(O-E)^2}{E}$$But your expected values will differ for each cell of the table:$$E_{(r,c)} = \frac{n_r n_c}N\\\begin{array}{c|cccc|c}&\rm Red&\rm Green&\rm Black&\rm Blue&\rm Total\\\hline\text{Survey A}&n_{(1,1)}&n_{(1,2)}&n_{(1,3)}&n_{(1,4)}&n_{(1,\rm all)}\\\text{Survey B}&n_{(2,1)}&n_{(2,2)}&n_{(2,3)}&n_{(2,4)}&n_{(2,\rm all)}\\\hline\rm Total&n_{(\rm all,1)}&n_{(\rm all,2)}&n_{(\rm all,3)}&n_{(\rm all,4)}&N\end{array}$$

In your case, your null hypothesis is that the proportions of each color will be similar for both surveys. Most hypothesis testers like to set a rule before performing their significance tests that they'll reject the null hypothesis model if there's less than a 5% chance of getting data that violates the model's expectations as much as or more than their sample data does if the model is true of the population from which the sample was randomly selected. It's not really necessary (and may even be inadvisable) to just dichotomize your attitude toward the null hypothesis model into a decision of whether to reject it wholesale, but that's the conventional approach, so it's probably what others would expect you to do.

It's worth noting that there's a Bayesian alternative, whereas the significance testing approach I've just described follows frequentist theory. One may also choose to be more interested in estimating the effect size and the level of confidence with which one can place that estimate within a confidence interval (or margin of error ×2) first and foremost, and consider the question of whether the margin of error on the side of zero includes the null hypothesis as a secondary concern. It's definitely not "crude" to prefer this approach. If anything, it's more crude to ignore your actual effect size estimate just because you're less than 95% confident that it would be at least as large if you repeated the study and the null is literally true (which it often isn't).

What effect size estimate? you might ask. (Briefly, since you didn't...) Cramér's $\phi$. For more on that, see "Chi squared test with expected frequencies coming from another observation".

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Excellent - thanks very much, Nick, for the comprehensive and informative answer. Now I've got some reading to do! $\endgroup$ – goblin-esque Jun 9 '14 at 6:35
  • $\begingroup$ We've all got reading to do. Statistics is incredibly tricky! If all I've said (and linked) here is enough (*), consider yourself lucky! And don't get hooked...you'll never be able to stop reading; not if you want to have much confidence in your results! At least, in my experience, my complacency about all the methodological assumptions was the first thing to go, and I've been reading ever since trying to get that sense of security back. Or maybe I'm just hooked on stats, or missing my calling as a "scared straight"-style motivational speaker... $\endgroup$ – Nick Stauner Jun 9 '14 at 7:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.