Making a measure of ranking accuracy Say I wanted to make a measure of how well a formula ranks a set of contestants in a running race. The inputs to the formula would be factors factors such as height, bodyweight, diet etc.
I had thought that the following would be reasonable. At the end of each race we take the absolute value of the difference between its rank (as the person finished in the race) and their predicted rank (as dictated by the formula). This would be summed for every person in the race. Finally this number would be divided by the number of people in the race - this last step is so that we can directly compare the ranking accuracy of large races to small ones. Call this number the "rank-accuracy".
I then decided to double check the last step - the "divide by the number of entrants" part. To do this I simulated random race results and random predictions and measured the average rank accuracy for a variety of race sizes. The results are as follows:
sz =  2 0.4992
sz =  3 0.8879
sz =  4 1.2500
sz =  5 1.6003
sz =  6 1.9424
sz =  7 2.2885
sz =  8 2.6230
sz =  9 2.9623
sz = 10 3.2983
sz = 11 3.6319
sz = 12 3.9764

As you can see my desire to make the rank-accuracy similar for all race sizes appears to have failed. It is clearly easier to get a low rank accuracy if the race size is small.
So my question is - what measure of rank accuracy could I use to make large and small races more directly comparable?
 A: It seems you are reinventing the wheel here. To compare the ranks given by your scoring function to the true ranks, you can just use Spearman correlation. This method simply converts your scores to numeric ranks, and computes the linear relationship with the true ranks. It penalizes large errors, so if your predictions have first and last place swapped, that will result in a worse association than if second and third place are swapped, for example. You can also compute a confidence interval on the Spearman correlation to determine if your scoring method is doing significantly better than random. Races with small N will have larger confidence intervals, since it is easier to rank only 2 items correctly by chance alone, but that's a feature and not a bug. The actual point estimate shouldn't be particularly affected by N, though - randomly generated scores will have near-zero correlation with the true ranks, regardless of N.
A: You may not need to "normalize" your rankings like you are doing it. Just avoid the average and division steps, and compare the ranks directly:
Let's assume you have 2 models, from which n ranks are generated by each model.
Then, each rank can be evaluated, say using Precision@K, NDCG@K etc, and only then you can measure the quality of each model by averaging its ranks. The compare the averages from each model, which gives you a first good analysis. You can also perform statistical analysis to check about how confident you think a model is better than the other, but for now, you are fine in comparing their means.
The term K, for those measurements mentioned, means the cut-off of interest, up to which position you are interested in evaluating. Other measures, such as MAP, evaluate the whole rank regardless of the cut-off.
A: I'm not entirely sure if I understand your question correctly, but if I visualise some worst case scenarios it will look like this (1st row = expected result, 2nd row = actual outcome, for 6 races size 2 to 7):
1   2
2   1   ->  2/2

1   2   3
3   2   1   -> 4/3

1   2   3   4
4   3   2   1   ->  8/4

1   2   3   4   5
5   4   3   2   1   ->  12/5

1   2   3   4   5   6
6   5   4   3   2   1   ->  18/6

1   2   3   4   5   6   7
7   6   5   4   3   2   1   ->  24/7

The arrows show the absolute rank difference/race size. As you can see, the absolute difference increases more than the size of the race. So a larger race can potentially have a higher average rank difference.
(specifically the relationship is: (n^2 +n) /2 - (ceil(n/2)) versus n, with n being the race size)
Normalise by maximum error
I think it's important to understand what you really want to normalise. Of course you can simply normalise using the maximum possible rank difference. So normalise so that size 2 maximum error equals size 7 maximum error.
Or in otherwords, state that 2/2 = 100% wrong, and 28/7 is also 100% wrong, and go from there.
Take nr of contestants incorrect into account?
Of course you can also look at number contestants that were ranked incorrectly. For instance in race size 3:
1   2   3
3   2   1   -> 4/3

but

1   2   3
3   1   2   -> also 4/3

In the first case only two contestants are ranked incorrectly, but in the second case all three are ranked incorrectly (although two of them "less incorrect").
Alternatively, use historical or current data
Alternatively, you can normalise using historical data, or your current results evenly. I.e. state that size2 0.4992 error equals your size12 3.9764 error (assuming you indeed consider that these are ranked equally good). Of course you need to be able to explain why you consider these equal, so you'll need some metric of an "equally good ranking"
Small number statistics
One thing to take into account is small number statistics. A race with size 2 has only two outcomes: 100% correct, or 100% wrong (assuming no split 1st place). So it might be difficult to compare the results if you do not use enough test cases.
