# How to measure the stability / consistency of a ranking

I am looking for the correct statistical method to answer the problem below. I have a method, but I am convinced there is a better way.

The scenario:

I have $N_x$ categorical independent variables. For each categorical variable $x$, there are $N_r$ repetitions, resulting in $N_r \times N_x$ continuous variables $y$. My goal is to establish a ranking of the $x$ variables according to $y$.

I would like to demonstrate that such a ranking is stable / consistent. I.e. the ideal result would be that the ranking is identical with infinite repetitions. The aim of this question is to be able to compare the consistency of different scenarios.

The brute force solution:

Combine each y (with differing x) to get every possible ranking. This gives $N_{r} ^ {N_{x}}$ possible rankings. For each pair of rankings, find the correlation coefficient (Spearman's rho, Kendall Tau, ...). Find the mean and standard deviation of these correlation coefficients.

This is obviously completely impractical with e.g. 10 repeats and 50 categorical variables. It is possible to sample the possible rankings, and then estimate the mean and standard deviation just of that sample, but that seems like throwing data away.

Alternatives

I've found How to measure the reliability of a consensus ranking (problem from Kemeny-Snell book) , but after reading the response I'm not really any wiser.

• Please, help us to understand your problem better. You have some categorical variables, let's say $N_x=3$ and you have categories apple, banana, cherry. You have some repetitions, say $N_r=10$. What would be your $y$? Weight? Are you going to proclaim something like apple$\succ$cherry$\succ$ banana ? Jun 19 '17 at 18:09
• Yes, I think that would be a reasonable example. The problem is (of course) more complicated. The categories are actually pairs of variables - for example, London/Paris, Paris/Brussels, Brussels/London. $y$ could be the journey time between them. But for the purposes of this questions, I'm not interested in the other possible questions (like "Does London have significantly slower journey times"), just establishing a ranking. So e.g. if I produce a ranking of journey times via train, is this more reliable than a ranking of journey times by car? Jun 19 '17 at 19:14

If you assume your $y$ to be normal with the same variance in each group, then ANOVA and t-tests can be used.