8
$\begingroup$

Four students sit a series of tests, each test contains 10 questions. Students are ranked 1-4 in each test depending upon who got the most questions correct.

                     Tom    Dick    Harry   Alice
   Test1 Score       9      5       6       8
   Test2 Score       8      3       7       4
   Test3 Score       7      6       9       8
   Test4 Score       9      2       5       7

Tom was ranked 1st in 3 of the 4 tests.

Test 5 is happening next week. How do I go about identifying the probability that Tom will be ranked first ? What is the probability that Alice will come first ?

While this is a small sample, I am interested in the general process that could apply to millions of test scores.

Thus: How do I predict the probability of an individual achieving a given performance rank for a future test based on previous performance on a set of tests for a set of individuals?

$\endgroup$
3
  • 1
    $\begingroup$ There are many possible models. You have to choose one. Are the test questions all indistinguishable? Are the performance of the students on each test question related? $\endgroup$ Commented Oct 10, 2012 at 1:19
  • $\begingroup$ Ok, so what are the various models I could use ? Presumeably, your questions will help decide which model is best to use ? All of the students would sit the same test with the same questions. If we say the questions are General Knowledge quiz type questions, the performance of each student on a particular question is related to their own general knowledge rather than any relation to the other students knowledge. $\endgroup$
    – Scott
    Commented Oct 10, 2012 at 7:20
  • $\begingroup$ So, some questions are easier for all students, while some questions are harder for all students? I'm just pointing out one thing which can affect the analysis of the results, so that you should include it if it is present. I'm not promising that this is the only thing missing. Is each student always competing with the same other 3 students? $\endgroup$ Commented Oct 10, 2012 at 7:38

2 Answers 2

0
$\begingroup$

There is one big caveat and several smaller ones. First an approximate answer for Tom ranking first is $p=$ 0.6315 with 95% confidence intervals of 0.6306 to 0.6324. Note, the confidence intervals are from how precisely I determined an answer, not from what the variability of the probability actually is. Now the gory details.

One cannot use Wilcoxon or other ranking methods, there are just too many ties. Thus, the Big Caveat: I assume that the ranker knows how to apportion tied scores exactly to award only one first place rank. I don't need to know how to do that, as follows.

I transformed the data into an approximately normal distribution, and to do this on a larger data set, it would have to be redone properly on that data. To do that, I took all the data, and tested it for normality. The mean and median were different and the tails were asymmetric, and the data looked vaguely like r-values. So, I took the $ArcTanh($test score$/10)$ (Fisher's transformation), which made the data a lot more normal. The reader should not take the transformation that I used to heart. One caveat is that it would not allow for a perfect score of 10, of which, for my quick and dirty approximation, there were none. A better distribution can only be found from analyzing more data. I then found the mean and standard deviation of that transformed data and did 1000 Monte Carlo simulations using inverse normal 1000 times. For each simulation, I counted when Tom had the best score. Now because I used real numbers and not integer scores, what would have been ties were adjudicated perfectly, without my knowing how to actually do that adjudication in practice. Alice's ranking first probability can be determined using the same methods.

Now there are probably 1000 ways of doing this same problem, and my humble suggestion has ifs ands and buts that may be avoided by handling the problem otherwise. My solution is only approximate, but, it is also quick and almost brain dead. I leave it up to others to suggest better methods.

$\endgroup$
1
$\begingroup$

A plan should be

  • consider a test as a 'match' between students
  • sort test in time order, and insert results in a rating system engine.

You can retrieve the probability that Tom will be ranked first in next test from ranking values.

If you're interested in use more than in development, rankade, our free ranking system for sports, games, and more, allows matches with both 2 and 3+ factions (as per your needs, while Elo and Glicko works just for one-on-one - here's a comparison). In addition, rankade has a weight feature (all tests have same impact?) that might refine your work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.