# Compare two unpaired samples, each with multiple proportions

I have two samples of 20 households, one sample from each of two different areas; the data for each household is a) total adults in household and 2) adults with college education.

Area 1

1           2         1
2           1         0
3           4         1


...

Area 2

1           3         1
2           2         2
3           4         2


... I am interested in the proportion of adults per household who have college education and I want to know if this proportion is significantly different between the two areas. My question is: what test to use?

I can't use a standard test of equality of proportions because these require just the two pooled proportions for each area, which is begging the question. I suspect there is a lot of variance at the household level in both number of adults and proportion of college education, so this needs taking into account.

This seems a case for simple logistic regression.

Here is an example in R.

set.seed(1)
# generate some data
dd$College[dd$Area==1] <- rbinom(rep(1,20), dd$Adults[dd$Area==1], 0.4)
dd$College[dd$Area==2] <- rbinom(rep(1,20), dd$Adults[dd$Area==2], 0.7)
dd$Area <- as.factor(dd$Area)


You can fit the model with the glm function providing as dependent variable two columns with the number of "successes" and "failures" (people with vs without college education)

# fit logistic model
dd$noCollege <- dd$Adults - dd\$College
m0 <- glm(cbind(College, noCollege) ~ Area, family=binomial, dd)


You can assess the effect of Area on the proportion college education by means of the Wald test provided when you apply the summary function to the fitted model object:

> summary(m0)

Call:
glm(formula = cbind(College, noCollege) ~ Area, family = binomial,
data = dd)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-2.1467  -0.5400   0.2320   0.9513   2.0587

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)  -0.5355     0.2570  -2.084  0.03720 *
Area2         1.1746     0.3827   3.069  0.00215 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 54.786  on 39  degrees of freedom
Residual deviance: 44.946  on 38  degrees of freedom
AIC: 92.664


Alternatively, you can also use a likelihood ratio test, by comparing the fitted model with a null model which assume no difference between Area:

> anova(update(m0, .~. -Area), m0,test="Chisq")
Analysis of Deviance Table

Model 1: cbind(College, noCollege) ~ 1
Model 2: cbind(College, noCollege) ~ Area
Resid. Df Resid. Dev Df Deviance Pr(>Chi)
1        39     54.786
2        38     44.946  1   9.8399 0.001708 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Update. It might be the case that the probability of college education is different for household with different number of adults. And the expected difference related to the two areas might also vary depending on the number of adults in the household. To account for that you can use a mixed-effect logistic regression, using the number of adults as grouping factor and including a random slope for Area:

> library(lme4)
> mm0 <- glmer(cbind(College, noCollege) ~ Area + (Area|Adults), family=binomial, dd)
> summary(mm0)
Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod']
Family: binomial  ( logit )
Formula: cbind(College, noCollege) ~ Area + (Area | Adults)
Data: dd

AIC      BIC   logLik deviance df.resid
98.7    107.1    -44.3     88.7       35

Scaled residuals:
Min      1Q  Median      3Q     Max
-1.7109 -0.5171  0.2312  0.7983  1.8327

Random effects:
Groups Name        Variance  Std.Dev. Corr
Area2       0.0007963 0.02822  -1.00
Number of obs: 40, groups:  Adults, 6

Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept)  -0.5339     0.2629  -2.031   0.0423 *
Area2         1.1748     0.3837   3.061   0.0022 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
(Intr)
Area2 -0.658


Notice that here I simulated the data using (for each area) the same probability of college education regardless of the number of adults. In fact in this example adding the random effects does not improve the fit. You can see it by comparing the AIC values of the two models: 92.7 for the simple logistic regression, 98.7 for the mixed effect regression, that is a difference of ~6 favouring the simple model with no random effects.

• This data might be more appropriately modelled with a mixed effect regression, as there are multiple adults per household. Not sure if you weren't aware of that, or just chose not to mention it because of the small sample size (which potentially renders even simple logistic regression difficult), but worth pointing out at least. Sep 27 '16 at 9:49
• Aha, yes glm to model both the elements of the proportion at the same time, that works, and yes there are random effects too. Sep 28 '16 at 8:37
• Right, one could use the number of adults as a grouping factor to account for possible differences in the probability of college education between large/small households. I'll update the answer Sep 29 '16 at 8:28

In first place, please notice that your sample is very small and that reduces the power of any test you use. That is, no test will find a significative difference unless the differences in the samples are very big.

From your question I'm not sure about which is your variable of interest, but I assume it's proportion of people with college education in the adults of the same household, that is, the ratio between "college" and "adults" variables in your sample.

The question "if this proportion is different" can have several meanings, too.

Since your sample is tiny and assumptions of normality wouldn't be realistic, the ubiquitous independent samples t-test wouldn't be a good choice. Furthermore it just tests for difference on means, not difference of distributions.

Therefore, you should use a non-parametric equivalent of t-test as Mann–Whitney U test which doesn't make a normality assumption and tests for difference of distributions.

However, if you want to test equality of the total number of college educated people among the total number of adults in each population (that is college/adults), then a proportion test could be useful - although your sample size is in the borderline of its usability.