# Students answer the same test twice; is MANOVA appropriate to test if the performance increased?

A teacher gives a 2-question quiz to a class of 30 students once before teaching the subject and another time at the end of the class. (The same exact 2 questions are given at the beginning and at the end of the class,i.e, Q11 is same as Q21 and Q12 is same as Q22). Each question is worth 1 point. The data looks like the following

name| Q11 |Q12 |Q21 |Q22
s1  |  .2 |.5  |.23 |.4
s2  |   0 |.2  |.4  |.4
...   ..  ..   ..  ..
s30 |  .15|.4  |.2  |.1


Can I apply MANOVA for repeated measures to this data? And how can I do it in R?

Clarification: The test actually has 10 questions. The sort of questions that I would like to answer are:

1. Are the performance on the first and second quiz different?
2. If there is a difference, which questions are responsible for the difference?

Assuming all you want to know is whether students scored better after instruction than before, then yes, you can use MANOVA to analyze the data. However, an easier method would be to use repeated measures ANOVA. Repeated measures ANOVA is a special case of MANOVA, so assuming you meet the assumptions of repeated measures ANOVA, you would be doing the same thing as MANOVA. The only caveat is that repeated measured ANOVA assumes your data is spherical (while the more general MANOVA does not). If it is not, then your results will likely be invalid.

While there are lots of tutorials out there on doing repeated measures ANOVA, most are used to find main effects and interactions in a classic ANOVA style. You have a specific question in mind, so it would be easiest to simply test that specific question: "Does the pretest score differ from the post test score?" This is equivalent to asking if $(Q21 + Q22) - (Q11 + Q12) > 0$

In R this is easily done by a simple linear model, and testing intercept.

dat$quiz.diff <- (dat$Q21 + dat$Q22) - (dat$Q11 + dat$Q12) summary(lm(quiz.diff~1,data=dat))  Other questions would likely be assessed using different contrasts between the variables. Note that while you are not using any special repeated measures functions, this is still a repeated measures ANOVA. You are simply doing some of the math yourself in making the contrast. Edit: I saw your response below where you say you want to know which questions caused the increase in quiz scores. I am still unclear as to exactly what the rest of your data looks like, so I'm just going to go on what you provided. If one question contributed more to the quiz score increase than the other, that means that the change in question 1 is significantly greater or less than the change in question 2. Or$(Q21 - Q11) > (Q22 - Q12)$which is equivalent to$(Q21 - Q11) - (Q22 - Q12) > 0$. So you would make a new variable with that contrast, and create a new linear model with that variable, testing the intercept of the model. • If you have more than 2 questions, and you want to test if an individual question improved more than other questions, it would make the most sense to me to see if the change in that question is greater than the average change in the other questions or$(Q21 - Q11) - AVG((Q22 - Q12),(Q23 - Q13), ...) > 0\$ Jul 8, 2015 at 18:53
• Are you essentially recommending to compute a difference score for each subject and then perform a one-sample t-test on these scores? Jul 8, 2015 at 19:27
• I'm not super familiar with the math on single-sample t-tests right now, so I cannot say for sure if it is the same as computing a single contrast value and testing the intercept in a GLM. If you are suggesting that the method may be invalid, you can find this method in Keppel & Wickens (2004) pp. 408-416; Judd, McClelland, and Ryan (2009) pp. 251-263, and I'm sure elsewhere. Jul 8, 2015 at 19:40
• I think the method is fine (+1). I was trying to indicate that what you suggested here is actually simpler than it might potentially seem (repeated measures ANOVA, hand-coded contrast... -- scary scary, but actually it's just a t-test). I am fairly sure that it's equivalent to a t-test, yes. Jul 8, 2015 at 22:13
• Yes, it's probably just laziness (and the way I was trained) but if something can be thought of as a GLM that's how I approach it since it's equivalent to a host of more specialized tests. Jul 9, 2015 at 13:22

I ended up using Multivariate Extensions of McNemar’s Test. Here is the paper http://www.stat.ufl.edu/~aa/articles/klingenberg_agresti_2006.pdf

Table 2 in this paper is exactly like my data (students) in the question.

If that's all the data you have, then there is nothing to model. For an ANOVA (or, really, any sort of similar model) you need an independent variable and you haven't got one. E.g boys vs. girls.

In addition, it's unclear what your data represents, how do the students get fractional scores on single questions?

If you really do have one, then you can use a (possibly nonlinear) multilevel model, but it would be much better if you had more than two time points and more than two questions

• Thanks Peter. The test actually has 10 questions. The sort of questions that I would like to answer are: 1-Are the performance on the first and second quiz different? 2-If there is a difference, which questions are responsible for the difference? Jul 8, 2015 at 18:00
• The same exact 2 questions are given at the beginning and at the end of the class,i.e, Q11 is same as Q21 and Q12 is same as Q22 Jul 8, 2015 at 18:14
• This isn't entirely true. If the research question is, "Did students quiz scores improve after the instruction?" then this is all the data you need. Its a basic repeated measures ANOVA problem (which is a special case of MANOVA). Jul 8, 2015 at 18:17
• Thanks le_andrew. It is repeated measures of several question. So it is multivariate. Am I right? So the first question, as you mentioned is: "Did students quiz scores improve after the instruction?" and the second question is: "If there is an improvement, which questions improved in their scores?" Jul 8, 2015 at 18:26

If you could organize your data as in:

| Moment | Student | NumberOfCorrectAnswers    |
|--------|---------|---------------------------|
| Before | 1       | 3                         |
| Before | 2       | 4                         |
| Before | 3       | 3                         |
| ...    | ...     | ...                       |
| After  | 1       | 8                         |
| After  | 2       | 7                         |
| ...    | ...     |                           |


This could be analyzed with an one-way anova. So in R you would use:

anova(NumberOfCorrectAnswers ~ Moment, data)


where data is a data frame with the columns above.

• Regular ANOVA would likely be invalid here because the observations are not independent (you have multiple observations from each person). It would only be valid if intra-class correlations are 0. Jul 8, 2015 at 18:19
• Walter, I want to know which questions had an improved scores, if there is any improvements. Jul 8, 2015 at 18:27
• Do you want separate analysis for each question? Jul 8, 2015 at 18:38