I have two groups: A (sample size = 20) and B (sample size = 6). Each of the 26 subjects takes a test with 25 items on it. The response for each item is binary (pass/fail). Thus, the data for each subject follows a binomial distribution(p, 25). I want to see if there is a difference in the two groups in terms of the proportion of items passed. The only method I can think of is to fit a logistic regression: group ~ p. However, the sample size in group B is too small in my opinion.

Another way would be compare the mean proportion in group A to the mean proportion in group B. However, the empirical distribution of p is heavily skewed right in both groups and there are many ties, so t-tests and Wilcoxon rank sum tests are off the table.


   prop.pass Group studentid
1       0.10     A Student01
2       0.00     B Student02
3       0.14     A Student03
4       0.05     A Student04
5       0.05     A Student06
6       0.00     B Student07
7       0.00     A Student08
8       0.33     A Student09
9       0.00     A Student10
10      0.52     A Student11
11      0.00     A Student12
12      0.52     A Student13
13      0.00     B Student14
14      0.55     A Student16
15      0.15     A Student18
16      0.15     A Student19
17      0.09     A Student20
18      0.00     A Student21
19      0.00     A Student22
20      0.00     A Student23
21      0.05     A Student24
22      0.00     A Student25
23      0.05     A Student26
24      0.00     B Student28
25      0.01     B Student05
26      0.05     B Student15
  • $\begingroup$ That's one hard test. A lot of students answered 0 questions correctly. $\endgroup$ – gung - Reinstate Monica May 4 '17 at 17:00
  • $\begingroup$ Actually, the way the data was given to me was mislabeled. Should have corrected it. It's proportion failed $\endgroup$ – nonclinbiostat May 4 '17 at 17:02
  • $\begingroup$ Well, that's better. Why does student01 have a proportion failed of 0.10 out of 25 questions? That's 2.5 missed. If they missed 2 it should be 0.08, & if 3 it would be 0.12. $\endgroup$ – gung - Reinstate Monica May 4 '17 at 17:04
  • $\begingroup$ Good eye. I wanted to keep the question as simple as possible, so I described the data as the investigator told me the study was conducted. I realized the same thing, so there's something misaligned between what I was told about the study design and the data. Can we just pretend for the time being that n=25 for all students? $\endgroup$ – nonclinbiostat May 4 '17 at 17:08

The sample size for B is not "too small" for logistic regression. Logistic regression does not make assumptions about sample sizes. You may have low power, though, or you may have other problems. The first thing I notice is that the data don't make sense in the context of your story (viz., the proportions don't equate to whole numbers of questions missed out of 25). As a result, software wouldn't let you fit a logistic regression model to these data. In addition, I doubt these are really distributed as a binomial. A binomial distribution is composed of a number of coin flips (metaphorically speaking), where each has an equal probability of being a success. That isn't likely to be the case with test questions: some will be harder and some will be easier. On the other hand, test questions are often given equal weight towards the test grade irrespective of that, so you may prefer to model it that way anyhow. Your other choice is to do something to allow for greater variance than the binomial expects, such as using the quasi-binomial distribution.

You are right that a t-test would not be appropriate here, but the Wilcoxon rank sum test does not require unskewed data, so it could be an acceptable choice as well. The rank sum test will implicitly assume that the questions are all equally hard, which is probably false, but if the test is scored that way, it might be acceptable nonetheless.

d = read.table(text="row   prop.pass Group studentid
1       0.10     A Student01
26      0.05     B Student15", header=T)
d = d[,-c(1,4)]
colnames(d)[1] = "prop.failed"

  ## R unhappy:
m1 = glm(prop.failed~Group, d, family=binomial, weights=rep(25, 26))
# Warning message:
# In eval(expr, envir, enclos) : non-integer #successes in a binomial glm!
# ...
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)    
# (Intercept)  -1.8362     0.1299 -14.140  < 2e-16 ***
# GroupB       -2.7589     0.8308  -3.321 0.000898 ***
# ...
# (Dispersion parameter for binomial family taken to be 1)
#     Null deviance: 163.54  on 25  degrees of freedom
# Residual deviance: 135.52  on 24  degrees of freedom
# AIC: 181.91
# ...

d$failed = round(d$prop.failed*25, digits=0)
d$passed = 25-d$failed
  ## no problems:
m2 = glm(cbind(passed, failed)~Group, d, family=quasibinomial)
# ...
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   1.8489     0.3265   5.663 7.85e-06 ***
# GroupB        3.1550     2.5320   1.246    0.225    
# ...
# (Dispersion parameter for quasibinomial family taken to be 6.263472)
#     Null deviance: 169.23  on 25  degrees of freedom
# Residual deviance: 138.96  on 24  degrees of freedom
# ...

wilcox.test(prop.failed~Group, d)
#   Wilcoxon rank sum test with continuity correction
# data:  prop.failed by Group
# W = 90, p-value = 0.06086
# alternative hypothesis: true location shift is not equal to 0

The second logistic regression model (the one that uses the rounded data in successes to failures format with the quasi-binomial distribution) is not significant because it is assuming more variability under the null. This would be my first choice, though. On the other hand, using the binomial distribution might be OK, as might the rank sum test. You do need to clarify your data, though, and make sure they are actually correct.

  • $\begingroup$ Thanks! Wouldn't the rank sum test be problematic with so many ties? $\endgroup$ – nonclinbiostat May 4 '17 at 17:56
  • $\begingroup$ @nonclinbiostat, it is relying on the normal approximation. I think the final answer here is that your data are ambiguous; that's often the way it is. $\endgroup$ – gung - Reinstate Monica May 4 '17 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.