How do I test to see if the demographic proportions between two unequally sized groups are statistically different?

Question

Sample of 102 broken into two groups based on a factor of interest

Group 1, $n=77$

Group 2, $n=25$

Trying to figure out how to test if the demographic (age, race/ethnicity, gender, education level, have kids, etc) break down is significantly different between groups.

I've calculated proportions and it seems like most of it will be fine, but there are a few that look like they could be significantly different.

I'm hoping to be able to run this in SPSS.

Answer 1

I have two points to make in answering this question.

Point the first
First, the z test for difference in proportions of two independent samples is pretty straightforward:

The null hypothesis is H$_{0}\text{: }p_{1} - p_{2} = 0$ (i.e. H$_{0}\text{: }p_{1} = p_{2}$), with H$_{\text{A}}\text{: }p_{1} - p_{2} \ne 0$.

$z = \frac{\hat{p}_{1}-\hat{p}_{2}}{\sqrt{\hat{p}\left(1-\hat{p}\right)\left[\frac{1}{n_{1}} + \frac{1}{n_{2}}\right]}}$,
where:
$\hat{p}_{1}$ and $\hat{p}_{1}$ are the sample proportions in group 1 and group 2;
$n_{1}$ and $n_{2}$ are the sample sizes in group 1 and group 2; and
$\hat{p}$ is the estimate of the sample means if H$_{0}$ is true, the best guess of which is simply the overall sample proportion (i.e. of all the data, ignoring which group an observation is from).

You might want to consider a continuity correction given that your combined sample size is 102. For example, Hauck and Anderson's (1986) correction gives:

$c_{\text{HA}} = \frac{1}{2\min{(n_{1},n_{2})}}$, and a redefined $s_{\hat{p}}$:

$s_{\hat{p}}= \sqrt{ \frac{\hat{p}_{1}(1-\hat{p}_{1})}{n_{1}-1} + \frac{\hat{p}_{2}(1-\hat{p}_{2})}{n_{2}-1}}$, so that

$z = \frac{\left|\hat{p}_{1} - \hat{p}_{2}\right| - c_{\text{HA}}}{\sqrt{ \frac{\hat{p}_{1}(1-\hat{p}_{1})}{n_{1}-1} + \frac{\hat{p}_{2}(1-\hat{p}_{2})}{n_{2}-1}}}$

The appropriate $p$-value for this $z$-statistic is then calculated or looked up in a table, and compared to $\alpha/2$ (two-tailed test).

Point the second
All differences are "statistically significant" given a large enough sample size. So a good idea is to decide beforehand what the smallest relevant difference in proportions is to you, and then look for evidence of such relevance. You find such evidence by combining the inferences from the test for difference just described, with a test for equivalence.

Suppose you decide beforehand that a meaningful difference in proportion for your purposes is on that is at least 0.05 (i.e. $|p_{1} - p_{2}| \ge 0.05$), then the corresponding test for equivalence of proportions for two independent groups is:

H$^{\text{-}}_{0}\text{: }|p_{1} - p_{2}| \ge 0.05$, which translates into two one-sided null hypotheses:

1. H$^{\text{-}}_{01}\text{: }p_{1} - p_{2} \ge 0.05$
   
2. H$^{\text{-}}_{02}\text{: }p_{1} - p_{2} \le -0.05$

These two one-sided null hypotheses can be tested with:

1. $z_{1} = \frac{0.05 - \left(\hat{p}_{1}-\hat{p}_{2}\right)}{\sqrt{\hat{p}\left(1-\hat{p}\right)\left[\frac{1}{n_{1}} + \frac{1}{n_{2}}\right]}}$, and
   
2. $z_{2} = \frac{\left(\hat{p}_{1}-\hat{p}_{2}\right)+0.05}{\sqrt{\hat{p}\left(1-\hat{p}\right)\left[\frac{1}{n_{1}} + \frac{1}{n_{2}}\right]}}$.

With a continuity correction $z_{1}$ and $z_{2}$ instead become:

1. $z_{1} = \frac{0.05 - \left(\hat{p}_{1}-\hat{p}_{2}\right) + c_{\text{HA}}}{\sqrt{ \frac{\hat{p}_{1}(1-\hat{p}_{1})}{n_{1}-1} + \frac{\hat{p}_{2}(1-\hat{p}_{2})}{n_{2}-1}}}$, and
   
2. $z_{2} = \frac{\left(\hat{p}_{1}-\hat{p}_{2}\right)+0.05-c_{\text{HA}}}{\sqrt{ \frac{\hat{p}_{1}(1-\hat{p}_{1})}{n_{1}-1} + \frac{\hat{p}_{2}(1-\hat{p}_{2})}{n_{2}-1}}}$.

If you reject both H$^{\text{-}}_{01}$ and H$^{\text{-}}_{02}$ (both tested at $\alpha$, not $\alpha/2$, and both tested with right tail rejection regions), then you can conclude that you have evidence of equivalence.

Finally... if you combine inference from tests of H$_{0}$ and H$^{\text{-}}_{0}$ (i.e. test for difference and test for equivalence), then you get one of the following possibilities:

1. reject H$_{0}$ and reject H$^{\text{-}}_{0}$: conclude trivial difference between proportions (i.e. yes there is a difference, but it's too small for you to care about);
   
2. reject H$_{0}$ and not reject H$^{\text{-}}_{0}$: conclude relevant difference between proportions (i.e. larger than 0.05);
   
3. not reject H$_{0}$ and reject H$^{\text{-}}_{0}$: conclude equivalence of proportions; or
   
4. not reject H$_{0}$ and not reject H$^{\text{-}}_{0}$: conclude indeterminate (i.e. underpowered tests).

References

Hauck, W. W. and Anderson, S. (1986). A comparison of large-sample con- fidence interval methods for the difference of two binomial probabilities. The American Statistician, 40(4):318–322.

Answer 2

I think you could use a Chi Square test of independence to answer this question. Chi Square tests of independence allow you to examine whether there is a relationship between two categorical variables (i.e. gender and group) or if the relationship between those two variables is the same as would be expected based on chance alone. This would work best if you have all of your data in one file, and a 2-level variable that indicates group.

I'd recommend running a series of chi-square analyses: 1 analysis for each demographic variable you're interested in. The second variable in the pair (for every analysis) will be your grouping variable. I'll include example syntax below. When you run the syntax, you'll first see output indicating the percentage composition of your groups. Then, if you scroll down, you want to look at the section of output that is titled "Chi-Square Tests." There, the Pearson Chi-Square value is the one you want to check out. To be significant, the column labeled "Asymp. Sig." needs to be 0.05 or smaller. If you get a significant result, that indicates that the two variables you are looking at are more related than would be expected by chance alone.

Here's some example SPSS syntax:

CROSSTABS /TABLES=Belonging BY Race /FORMAT=AVALUE TABLES /STATISTICS=CHISQ PHI /CELLS=COUNT ROW COLUMN TOTAL /COUNT ROUND CELL.

Here is the same syntax, but where you would place your values is indicated with Variable 1 and GroupingVariable.

CROSSTABS /TABLES=Variable1 BY GroupingVariable /FORMAT=AVALUE TABLES /STATISTICS=CHISQ PHI /CELLS=COUNT ROW COLUMN TOTAL /COUNT ROUND CELL.