To preface, I'm a stat noob programmer, so sorry if this is a basic question.

I have multiple (300+) exams that I had a variety of users take. The users could decide which exams to take. All the exams had different number of exam-takers (from around 30 to around 10000), different variances, and different means. The data is anonymized (I simply have a list of scores). I would like to order the exams from easiest to hardest.

I'm assuming that all the exam-takers are of relatively similar level (although if this is an assumption we can relax, that would be ideal) and this is a normal distribution. Through a little bit of googling, I found that I could calculate a sample mean and sample standard deviation from the data and use these 2 values (along with n to find the t statistic). However, I am unclear as to how to proceed.

I imagine there will be some hypothesis test to see which is larger. However, this seems to be computationally inefficient. What is the best way to do this so that I may rank these exams from easiest to hardest (ie. without comparing each one on a one to one basis)?

I was thinking I could some sort of Bayesian system as described in these 2 similar questions (1, 2), although I'm not sure how I would construct such a system. Would this work? This solution, while easy, seems like it wouldn't work given the scale of numbers (30ish on the low end vs 10000ish on the high end, so how many extra "balancing scores" does one add); I was also thinking of doing this, but also scaling it up by adding 10, instead of the 3 suggested by the linked question (where would I draw the line on number of scores to add?), exam scores of 50% (why 50% instead of 0%?). This question also proposes a more rigorous approach of cross validation, but I don't fully understand how this would be implemented.

  • 3
    $\begingroup$ You cannot relax your (very strong) assumption that all exam-takers have the same capabilities, because the outcomes on these self-selected exams tell us as much about who took them as they do about the exams themselves. Given this, your enterprise looks futile. $\endgroup$ – whuber Jul 10 '20 at 12:31
  • $\begingroup$ @whuber Assuming that I'm willing to accept the selection bias, how would this be done? $\endgroup$ – lwl59438cuoly Jul 10 '20 at 15:05
  • 2
    $\begingroup$ People frequently ask this question under different guises, such as how to rank Amazon star ratings. My answer always has been that there's no one method because you need to make a tradeoff between the estimated difficulty and the uncertainty in that estimate; and such trade-offs are a matter for you, not the software, to decide. Although you can find loads of blogs advocating formulas based on confidence limits and the like, they exist only because most people don't seem to recognize this fundamental arbitrariness in their solutions. Bayesian approaches might be appealing. $\endgroup$ – whuber Jul 10 '20 at 15:17
  • $\begingroup$ @whuber Is there anything I can look up to point me in the correct direction so that I may be able to read up on it and decide myself? If you have time, may I suggest writing an answer, highlighting the assumptions you have made so this can be easily understood? I'm also unclear on what you mean by the "tradeoff between the estimated difficulty and the uncertainty in that estimate". Is the estimated difficulty the computational difficulty? How would that go up with a lower level of uncertainty? $\endgroup$ – lwl59438cuoly Jul 10 '20 at 15:23
  • 1
    $\begingroup$ I use "difficulty" as you do: the degree of difficulty of the exam. This subject is under the purview of multi-attribute decision making. See en.wikipedia.org/wiki/Multi-attribute_utility for instance. $\endgroup$ – whuber Jul 10 '20 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.