My study consists of three treatments. One treatment group has 8 people and the other two 11.

Each person in the treatment group received three questions and I want to compare how many they answered correctly with the other two groups.

How would I do that? Should I calculate the percentage of people that got each question correctly and then do an analysis of variance (ANOVA)? Somehow that doesn't make sense to me.

  • 2
    $\begingroup$ So the outcome is essentially whether each person answered zero, one, two or three questions correctly? $\endgroup$ – James Stanley May 17 '15 at 21:41
  • $\begingroup$ So, each person in each treatment group recieved three questions? $\endgroup$ – Aghila May 17 '15 at 22:06
  • $\begingroup$ Each person in each treatment group receive three questions. And the outcome is how many questions each person answered correctly. $\endgroup$ – Tracy May 17 '15 at 22:41
  • $\begingroup$ I'm a bit confused with the design. Are you trying to make a one-factor design, where the factor has four levels: control, treatment 1, treatment 2 etc? But it makes no sense because you didn't apply any "treatment" to those groups, you were asking them questions.... $\endgroup$ – SmallChess May 18 '15 at 1:20
  • $\begingroup$ Possibly poisson regression may also be useful here: theanalysisfactor.com/… $\endgroup$ – rnso May 20 '15 at 11:37

Since your response is ordinal, doing any ANOVA or chi-squared test will lose the trend of the outputs. Consider doing a Cumulative Logit Model where multiple logits are formed of cumulative probabilities.

$$ \begin{align} P(Y \le j | x) &= \pi_1(x) + ... +\pi_j(x), \quad j=1, ..., J\\ logit\big[P(Y \le j | x)\big] &= \frac{P(Y \le j | x)}{1-P(Y \le j | x)}\\ &= \frac{\pi_1(x) + ... +\pi_j(x)}{\pi_{j+1}(x) + ... +\pi_J(x)} \end{align} $$

Finally we assume the same effect $\beta$ for all models and and look at proportional odds in a single model.

$$ logit\big[P(Y \le j |\textbf{x})\big] = \alpha_j + \beta^T\textbf{x}, \quad j=1,...,J-1 $$

In this case, you would have a reference group and two $x$'s that represent the two other groups

$$ logit\big[P(Y \le j |\textbf{x})\big] = \alpha_j + \beta_1x_1 + \beta_2x_2 $$

Model fit is checked by a "Score Test" and should be outputted by your software. The Score test checks against more complicated models for a better fit.

Finally, interpreting the results is straight forward by moving the logit to the other side

$$ P(Y \le j |\textbf{x}) = \frac{e^{\alpha_j + \beta^T\textbf{x}}}{1+e^{\alpha_j + \beta^T\textbf{x}}} $$

There are lots of more references on the internet. Agresti's Categorial Data Analysis is a great book for this which contain many alteratives if the this model doesn't fit.

If you want to stay simpler, consider doing a Kruskal-Wallis test, which is a non-parametric version of ANOVA. Like most non-parametric tests, it uses ranks instead of actual values and is not exact if there are ties. Like ANOVA, it will compare all three groups together. While it doesn't require the data to be normally distributed, it does require the data to have approximately the same shape.

If the null hypothesis test is rejected, then Dunn's test will help figure out which pairs of groups are different. It is also based on ranks,

  • 1
    $\begingroup$ Maybe I misunderstand, but why would you call these data ordinal? You can meaningfully take differences ("person A got one more answer correct than person B") and also ratios ("person A scored twice as many correct answers than person B"). $\endgroup$ – A. Donda Jul 29 '15 at 3:28
  • $\begingroup$ I agree with the comment, that these data don't need to be treated as ordinal, but I think using KW and Dunn test (1964) would be a simple and applicable approach. $\endgroup$ – Sal Mangiafico Nov 3 at 15:42

I don't think you should use ANOVA because the normality is not satisfied. Furthermore, your dependent variable is not continuous.

Mann-Whitney U test will give you what you want. Your dependent variable can be ordered (ordinal scale). Say, if your first group performs much better than the other group, you might have something like this:

1 1 1 1 1 2 1 1 2 2 2 2

The samples are ranked according to the number of questions answered correctly. In this example, group 1 answers much better than group 2.


Since it is a count data, poisson regression can also be applied here:

> dd
   grp value
1    x     3
2    x     3
3    x     3
4    x     1
5    x     2
6    x     3
7    x     2
8    x     1
9    y     3
10   y     3
11   y     3
12   y     3
13   y     1
14   y     3
15   y     3
16   y     2
17   y     2
18   y     3
19   y     3
20   z     2
21   z     3
22   z     1
23   z     2
24   z     1
25   z     1
26   z     2
27   z     3
28   z     1
29   z     2
30   z     3

> summary(glm(value~grp, data=dd, family='poisson'))

glm(formula = value ~ grp, family = "poisson", data = dd)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.1550  -0.4095   0.1422   0.2191   0.7281  

            Estimate Std. Error z value Pr(>|z|)    
(Intercept)   0.8109     0.2357   3.440 0.000581 ***
grpy          0.1585     0.3001   0.528 0.597414    
grpz         -0.1643     0.3212  -0.512 0.608989    
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for poisson family taken to be 1)

    Null deviance: 9.7656  on 29  degrees of freedom
Residual deviance: 8.4788  on 27  degrees of freedom
AIC: 94.266

Number of Fisher Scoring iterations: 4

This gives difference of y and z from x. Levels in grp variable can be changed for difference with respect to y or z.

  • 1
    $\begingroup$ I don't think Poisson is appropriate; nobody can get 4 or more. If you regarded all three questions as equally hard to answer correctly, you might use a binomial model; alternatively, if data were split by question and question was a factor, you could again use a binomial model. In the absence of either you might use a quasi binomial model. $\endgroup$ – Glen_b Jun 27 '15 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.