I'm working on an adaptive learning platform with some friends for a project that helps students to prepare their university admission test.

We provide different kind of practice sessions and we have some millions of answers given. Each question has 5 possible choices and we gathered the following data for each attempt made at answering a question:

  • Question ID
  • Answer ID
  • Outcome of the attempt (correct / incorrect / skipped)
  • Time taken to answer

My question is: how can I use the entire data set to rate the difficulty of a question?

My idea was to normalise each data removing answers given in too less or too much time and judge the difficulty using a float number between 0 (very easy) and 1 (very difficult). But how?

  • $\begingroup$ This is difficult to follow. Can you supply more background? $\endgroup$ – Michael R. Chernick Apr 13 '17 at 16:41
  • $\begingroup$ see the answer below. IRT is what you want. in IRT, difficulty is defined at the ability an individual requires in order to have a 50% chance of answering the item correctly. $\endgroup$ – faustus Apr 13 '17 at 19:41

This is problem that occurs very often in educational research. Google for "Item Response Theory". It is a family of models designed exactly for this purpose, and more advanced models of this family (like PL2 and PL3) provide even more information about questions than just the difficulty level.

This is very widely described approach, for example in PISA assessment test questionnaires are scaled with this approach. There are also free packages to R that allow to fit suitable models (mirt package for example). The most important issue to consider is, if there is just one trait that you are measuring with this test or there are more traits. If there is one dimension you want to measure, the problem is quite easy, if there are more dimensions it can be difficult.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.