How to predict the probability of a performance rank for a test taker on a future test based on previous test scores?

Question

Four students sit a series of tests, each test contains 10 questions. Students are ranked 1-4 in each test depending upon who got the most questions correct.

                     Tom    Dick    Harry   Alice
   Test1 Score       9      5       6       8
   Test2 Score       8      3       7       4
   Test3 Score       7      6       9       8
   Test4 Score       9      2       5       7

Tom was ranked 1st in 3 of the 4 tests.

Test 5 is happening next week. How do I go about identifying the probability that Tom will be ranked first ? What is the probability that Alice will come first ?

While this is a small sample, I am interested in the general process that could apply to millions of test scores.

Thus: How do I predict the probability of an individual achieving a given performance rank for a future test based on previous performance on a set of tests for a set of individuals?

Answer 1

There is one big caveat and several smaller ones. First an approximate answer for Tom ranking first is $p=$ 0.6315 with 95% confidence intervals of 0.6306 to 0.6324. Note, the confidence intervals are from how precisely I determined an answer, not from what the variability of the probability actually is. Now the gory details.

One cannot use Wilcoxon or other ranking methods, there are just too many ties. Thus, the Big Caveat: I assume that the ranker knows how to apportion tied scores exactly to award only one first place rank. I don't need to know how to do that, as follows.

I transformed the data into an approximately normal distribution, and to do this on a larger data set, it would have to be redone properly on that data. To do that, I took all the data, and tested it for normality. The mean and median were different and the tails were asymmetric, and the data looked vaguely like r-values. So, I took the $ArcTanh($test score$/10)$ (Fisher's transformation), which made the data a lot more normal. The reader should not take the transformation that I used to heart. One caveat is that it would not allow for a perfect score of 10, of which, for my quick and dirty approximation, there were none. A better distribution can only be found from analyzing more data. I then found the mean and standard deviation of that transformed data and did 1000 Monte Carlo simulations using inverse normal 1000 times. For each simulation, I counted when Tom had the best score. Now because I used real numbers and not integer scores, what would have been ties were adjudicated perfectly, without my knowing how to actually do that adjudication in practice. Alice's ranking first probability can be determined using the same methods.

Now there are probably 1000 ways of doing this same problem, and my humble suggestion has ifs ands and buts that may be avoided by handling the problem otherwise. My solution is only approximate, but, it is also quick and almost brain dead. I leave it up to others to suggest better methods.

Answer 2

A plan should be

• consider a test as a 'match' between students
• sort test in time order, and insert results in a rating system engine.

You can retrieve the probability that Tom will be ranked first in next test from ranking values.

If you're interested in use more than in development, rankade, our free ranking system for sports, games, and more, allows matches with both 2 and 3+ factions (as per your needs, while Elo and Glicko works just for one-on-one - here's a comparison). In addition, rankade has a weight feature (all tests have same impact?) that might refine your work.