[Note: I updated the question after getting comments, thanks!]

I am currently trying to find out if there is a significant difference in performance for different kinds of questions in a subject, but I have trouble finding the right way to analyze the data.

I have exams for about 10 years, with two different kinds of questions (type A and B), and wonder whether students perform better on type A than type B. The students get between 0 and 100 points on each question, so the mean score for each type of question is my response variable. All students answer all questions on each exam. The problem is further complicated by the fact that two different tutors make the questions. Thus difference in score might be due to one tutor making harder questions than the other. One tutor makes questions as type A more often than the other, so this might affect the result.

So for each question, I know the score all students got, which tutor made it, and which category (type A or B) it belongs to. Whether there is a difference between performance on type A and B is the interest here.

Example: Question 1 on the 2013 exam was type A and made by tutor 1. I know the scores from 30 students, so I can calculate the mean score and variance.

At first I thought I might just put all questions of type A in one group, and all questions of type B in the other group and perform a paired t-test, but that does not take into account that there are different groups of students over the years, and the questions have different variances (some questions are answered almost correctly by everyone, but most are not). In addition, there is the possibility that one tutor makes harder questions than the other, regardless of type.

EDIT: I think using a generalized mixed model might work, if I use tutor and type as covariates, and student as random intercept, and see if the coefficient for type is significant. Perhaps I should also included year as covariate? As the score is between 0 and 100, I am not quite sure which model I should fit though. A beta model would have been nice (just letting the score be between 0 and 1 instead), but is that possible?

Any thoughts on how I might approach this? I am very grateful for all help!

  • $\begingroup$ You imply that you only have summary data and the the detailed responses. Is that correct? $\endgroup$ – Harlan Nelson Feb 24 '18 at 16:00
  • $\begingroup$ you may indiicate definitions, perspective and what are your objectives. $\endgroup$ – Subhash C. Davar Feb 24 '18 at 16:11
  • $\begingroup$ Thanks for replying! I think I might have been unclear as I tried to keep the question short. I have detailed data, so that is no problem. So for all students, I have all scores on all problems. My main goal is to find out whether questions of type A are harder than those of type B. I think using the mean score on questions of type A and see if it is different than questions of type B is what I want to do, but if possible correct for the fact that most questions of one type is made by one tutor. I am however not sure which test to use. $\endgroup$ – SummerRed Feb 25 '18 at 18:36
  • $\begingroup$ For example I might have two questions of type A on the 2013 exam, where one has scores with low variance and the other has scores with high variance. The score on questions of type B will also be dependent on the score on questions of type A somehow, as the same students answered them (the questions do not depend on each other). So for one year I thought I might use a paired t-test, but I am not sure if that is ok when the questions have different variance. I think looking at the mean score for all years might be too simplified? $\endgroup$ – SummerRed Feb 25 '18 at 18:42
  • $\begingroup$ I think you also benefit from an estimate of the student ability... the same student is not seeing both question types? $\endgroup$ – seanv507 Feb 26 '18 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.