0
$\begingroup$

I'm working on a study where I am comparing how students did on three different exams. By the end of each exam, as always, students were assigned a final grade based on the percentage of questions they got right (80%, 83%, 73%, etc.) Each exam, however, had a different number of questions.

If I want to compare how the class did overall across the three exams, can I use a Repeated measures ANOVA, as someone (above me) has suggested?

I've read different things, including many saying it's not appropriate to use Repeated measures ANOVA with percentages. But I can't see how using the raw data would work, since, as I stated, each test had a different number of questions. Wouldn't that make it "comparing DIFFERENT variables," whereas a comparison of the percentages would actually make it a comparable measure of the SAME variable? Or should I ignore what's been suggested, as used a different measurement (I had used separate paired-sample t-tests, but was told I should do Repeated Measures ANOVA instead, as it'd allow me to compare all three samples at once)?

While I've tried running a Repeated measures ANOVA on the raw data, the profile plot, for example, shows a mathematically correct outcome, obviously, but one that does not reflect the outcome (eg On Exam 1, students got, on average, 3.45 of the 7 questions right; on Exam 2, they got 2.60 of the 5 questions right, and on exam 3, they got 2.14 of the 3 questions right. The plot, then, would show 3.45 - 2.60 - 2.14 -> therefore, a continuous downslope. However, if you look at the mean percentage, it's actually 49.3% - 52.0% - 71.3% -> therefore, continuously going up). Again, does that matter?

Advice as always would be welcomed! Thanks!

Improve this question
$\endgroup$
2
  • $\begingroup$ An important thing about most statistical procedures is that they don't care where the numbers came from. What matters is the population distribution. $\endgroup$
    – David Lane
    May 22, 2017 at 17:55
  • $\begingroup$ It is clear (obvious, whatever) that your analysis should compensate for the total size of each exam. Not to do so is absurd. The notion that you can't analyze percentages with ANOVA is both unhelpful and incorrect. Do the analysis; look at the residuals. $\endgroup$
    – David Smith
    May 22, 2017 at 18:29

1 Answer 1

Reset to default
1
$\begingroup$

One way to look at it is that "percentage" is actually exp(log(count) - log(total)). If you do a mixed-effects glm of Class vs. Correct Answers with a poisson distribution, offset for log of Total Questions (since poisson link is log), and random intercepts for "student", this could handle it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.