I am looking for the correct statistical method to answer the problem below. I have a method, but I am convinced there is a better way.

The scenario:

I have $N_x$ categorical independent variables. For each categorical variable $x$, there are $N_r$ repetitions, resulting in $N_r \times N_x$ continuous variables $y$. My goal is to establish a ranking of the $x$ variables according to $y$.

I would like to demonstrate that such a ranking is stable / consistent. I.e. the ideal result would be that the ranking is identical with infinite repetitions. The aim of this question is to be able to compare the consistency of different scenarios.

The brute force solution:

Combine each y (with differing x) to get every possible ranking. This gives $N_{r} ^ {N_{x}}$ possible rankings. For each pair of rankings, find the correlation coefficient (Spearman's rho, Kendall Tau, ...). Find the mean and standard deviation of these correlation coefficients.

This is obviously completely impractical with e.g. 10 repeats and 50 categorical variables. It is possible to sample the possible rankings, and then estimate the mean and standard deviation just of that sample, but that seems like throwing data away.


I've found How to measure the reliability of a consensus ranking (problem from Kemeny-Snell book) , but after reading the response I'm not really any wiser.

  • $\begingroup$ Please, help us to understand your problem better. You have some categorical variables, let's say $N_x=3$ and you have categories apple, banana, cherry. You have some repetitions, say $N_r=10$. What would be your $y$? Weight? Are you going to proclaim something like apple$\succ$cherry$\succ$ banana ? $\endgroup$ – Karel Macek Jun 19 '17 at 18:09
  • $\begingroup$ Yes, I think that would be a reasonable example. The problem is (of course) more complicated. The categories are actually pairs of variables - for example, London/Paris, Paris/Brussels, Brussels/London. $y$ could be the journey time between them. But for the purposes of this questions, I'm not interested in the other possible questions (like "Does London have significantly slower journey times"), just establishing a ranking. So e.g. if I produce a ranking of journey times via train, is this more reliable than a ranking of journey times by car? $\endgroup$ – Ollyver Jun 19 '17 at 19:14

For this purpose, you can use Kruskal-Wallis to test if there is any ranking that can be considered. https://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance

If it is the case, you can use Dunn's test for pair-wise comparisons.

These comparisons can form the ranking/ordering you expect.

If you assume your $y$ to be normal with the same variance in each group, then ANOVA and t-tests can be used.

Note that the ranking/ordering may not be linear as some pairwise comparison may exhibit differences that are not statistically significant, as e.g. Bob and Jane in the following example: http://www.quality-assurance-solutions.com/images/ANOVA-3.jpg

| cite | improve this answer | |
  • 1
    $\begingroup$ doyenne.com appears to now be a clothing website. $\endgroup$ – Ollyver Jun 20 '17 at 6:04
  • 1
    $\begingroup$ @Ollyver Updated the link to dunntest. :) $\endgroup$ – Alexis Oct 11 '17 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.