# Students answer the same test twice; is MANOVA appropriate to test if the performance increased?

A teacher gives a 2-question quiz to a class of 30 students once before teaching the subject and another time at the end of the class. (The same exact 2 questions are given at the beginning and at the end of the class,i.e, Q11 is same as Q21 and Q12 is same as Q22). Each question is worth 1 point. The data looks like the following

name| Q11 |Q12 |Q21 |Q22
s1  |  .2 |.5  |.23 |.4
s2  |   0 |.2  |.4  |.4
...   ..  ..   ..  ..
s30 |  .15|.4  |.2  |.1


Can I apply MANOVA for repeated measures to this data? And how can I do it in R?

Clarification: The test actually has 10 questions. The sort of questions that I would like to answer are:

1. Are the performance on the first and second quiz different?
2. If there is a difference, which questions are responsible for the difference?

Assuming all you want to know is whether students scored better after instruction than before, then yes, you can use MANOVA to analyze the data. However, an easier method would be to use repeated measures ANOVA. Repeated measures ANOVA is a special case of MANOVA, so assuming you meet the assumptions of repeated measures ANOVA, you would be doing the same thing as MANOVA. The only caveat is that repeated measured ANOVA assumes your data is spherical (while the more general MANOVA does not). If it is not, then your results will likely be invalid.

While there are lots of tutorials out there on doing repeated measures ANOVA, most are used to find main effects and interactions in a classic ANOVA style. You have a specific question in mind, so it would be easiest to simply test that specific question: "Does the pretest score differ from the post test score?" This is equivalent to asking if $(Q21 + Q22) - (Q11 + Q12) > 0$

In R this is easily done by a simple linear model, and testing intercept.

dat$quiz.diff <- (dat$Q21 + dat$Q22) - (dat$Q11 + dat$Q12) summary(lm(quiz.diff~1,data=dat))  Other questions would likely be assessed using different contrasts between the variables. Note that while you are not using any special repeated measures functions, this is still a repeated measures ANOVA. You are simply doing some of the math yourself in making the contrast. Edit: I saw your response below where you say you want to know which questions caused the increase in quiz scores. I am still unclear as to exactly what the rest of your data looks like, so I'm just going to go on what you provided. If one question contributed more to the quiz score increase than the other, that means that the change in question 1 is significantly greater or less than the change in question 2. Or$(Q21 - Q11) > (Q22 - Q12)$which is equivalent to$(Q21 - Q11) - (Q22 - Q12) > 0$. So you would make a new variable with that contrast, and create a new linear model with that variable, testing the intercept of the model. • If you have more than 2 questions, and you want to test if an individual question improved more than other questions, it would make the most sense to me to see if the change in that question is greater than the average change in the other questions or$(Q21 - Q11) - AVG((Q22 - Q12),(Q23 - Q13), ...) > 0\$ – le_andrew Jul 8 '15 at 18:53
• Are you essentially recommending to compute a difference score for each subject and then perform a one-sample t-test on these scores? – amoeba says Reinstate Monica Jul 8 '15 at 19:27
• I'm not super familiar with the math on single-sample t-tests right now, so I cannot say for sure if it is the same as computing a single contrast value and testing the intercept in a GLM. If you are suggesting that the method may be invalid, you can find this method in Keppel & Wickens (2004) pp. 408-416; Judd, McClelland, and Ryan (2009) pp. 251-263, and I'm sure elsewhere. – le_andrew Jul 8 '15 at 19:40
• I think the method is fine (+1). I was trying to indicate that what you suggested here is actually simpler than it might potentially seem (repeated measures ANOVA, hand-coded contrast... -- scary scary, but actually it's just a t-test). I am fairly sure that it's equivalent to a t-test, yes. – amoeba says Reinstate Monica Jul 8 '15 at 22:13
• Yes, it's probably just laziness (and the way I was trained) but if something can be thought of as a GLM that's how I approach it since it's equivalent to a host of more specialized tests. – le_andrew Jul 9 '15 at 13:22

I ended up using Multivariate Extensions of McNemar’s Test. Here is the paper http://www.stat.ufl.edu/~aa/articles/klingenberg_agresti_2006.pdf

Table 2 in this paper is exactly like my data (students) in the question.

If that's all the data you have, then there is nothing to model. For an ANOVA (or, really, any sort of similar model) you need an independent variable and you haven't got one. E.g boys vs. girls.

In addition, it's unclear what your data represents, how do the students get fractional scores on single questions?

If you really do have one, then you can use a (possibly nonlinear) multilevel model, but it would be much better if you had more than two time points and more than two questions

• Thanks Peter. The test actually has 10 questions. The sort of questions that I would like to answer are: 1-Are the performance on the first and second quiz different? 2-If there is a difference, which questions are responsible for the difference? – pmjn6 Jul 8 '15 at 18:00
• The same exact 2 questions are given at the beginning and at the end of the class,i.e, Q11 is same as Q21 and Q12 is same as Q22 – pmjn6 Jul 8 '15 at 18:14
• This isn't entirely true. If the research question is, "Did students quiz scores improve after the instruction?" then this is all the data you need. Its a basic repeated measures ANOVA problem (which is a special case of MANOVA). – le_andrew Jul 8 '15 at 18:17
• Thanks le_andrew. It is repeated measures of several question. So it is multivariate. Am I right? So the first question, as you mentioned is: "Did students quiz scores improve after the instruction?" and the second question is: "If there is an improvement, which questions improved in their scores?" – pmjn6 Jul 8 '15 at 18:26

If you could organize your data as in:

| Moment | Student | NumberOfCorrectAnswers    |
|--------|---------|---------------------------|
| Before | 1       | 3                         |
| Before | 2       | 4                         |
| Before | 3       | 3                         |
| ...    | ...     | ...                       |
| After  | 1       | 8                         |
| After  | 2       | 7                         |
| ...    | ...     |                           |


This could be analyzed with an one-way anova. So in R you would use:

anova(NumberOfCorrectAnswers ~ Moment, data)


where data is a data frame with the columns above.

• Regular ANOVA would likely be invalid here because the observations are not independent (you have multiple observations from each person). It would only be valid if intra-class correlations are 0. – le_andrew Jul 8 '15 at 18:19
• Walter, I want to know which questions had an improved scores, if there is any improvements. – pmjn6 Jul 8 '15 at 18:27
• Do you want separate analysis for each question? – Peter Flom - Reinstate Monica Jul 8 '15 at 18:38