I'm analyzing results of undergraduate exams that range from 60 to 100 multiple choice questions. The number of students doing any given exam ranges from ~70 to ~1000. In all cases students select from four options: A, B, C, and D.

The purpose of the analysis is to suggest changes that may be applied to the exam analysis in future years, for example questions that should be altered or distractor options that should be altered. Another purpose is to identify questions that are so defective that they should be removed from student scores for the present year.

I adapted the R code given in response to this question to do an item analysis. At present I am able to calculate:

Proportion correct (difficulty of the question)
Point-Biserial correlation, i.e. the correlation between score on the item and total score on the exam (discrimination power of question)
Mean total on the exam for people who got this question correct
Mean total on the exam for people who got this question incorrect
Number of students who got this question correct
Number of students who didn't do this question (usually 0 or close to 0, since there's no penalty for an incorrect guess)
Breakdown of the number of students who answered A, B, C, D

I've noticed that some online guides calculate a reliability coefficient for the exam, e.g. http://www.washington.edu/oea/services/scanning_scoring/scoring/item_analysis.html. Some guides suggest to calculate the KR20, while others suggest calculate Cronbach's alpha. How important is this, relative to the elements I have already mentioned? Is KR20 more appropriate for this context or Cronbach's alpha, or something else?

Have I missed any other important components of an MCQ exam item analysis, or included any superfluous components?

My goal is to balance providing useful information for the exam-writers without overloading them with too much output.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.