3
$\begingroup$

I am looking for the correct statistical method to answer the problem below. I have a method, but I am convinced there is a better way.

The scenario:

I have $N_x$ categorical independent variables. For each categorical variable $x$, there are $N_r$ repetitions, resulting in $N_r \times N_x$ continuous variables $y$. My goal is to establish a ranking of the $x$ variables according to $y$.

I would like to demonstrate that such a ranking is stable / consistent. I.e. the ideal result would be that the ranking is identical with infinite repetitions. The aim of this question is to be able to compare the consistency of different scenarios.

The brute force solution:

Combine each y (with differing x) to get every possible ranking. This gives $N_{r} ^ {N_{x}}$ possible rankings. For each pair of rankings, find the correlation coefficient (Spearman's rho, Kendall Tau, ...). Find the mean and standard deviation of these correlation coefficients.

This is obviously completely impractical with e.g. 10 repeats and 50 categorical variables. It is possible to sample the possible rankings, and then estimate the mean and standard deviation just of that sample, but that seems like throwing data away.

Alternatives

I've found How to measure the reliability of a consensus ranking (problem from Kemeny-Snell book) , but after reading the response I'm not really any wiser.

Improve this question
$\endgroup$
3
  • $\begingroup$ Please, help us to understand your problem better. You have some categorical variables, let's say $N_x=3$ and you have categories apple, banana, cherry. You have some repetitions, say $N_r=10$. What would be your $y$? Weight? Are you going to proclaim something like apple$\succ$cherry$\succ$ banana ? $\endgroup$
    – Karel Macek
    Commented Jun 19, 2017 at 18:09
  • $\begingroup$ Yes, I think that would be a reasonable example. The problem is (of course) more complicated. The categories are actually pairs of variables - for example, London/Paris, Paris/Brussels, Brussels/London. $y$ could be the journey time between them. But for the purposes of this questions, I'm not interested in the other possible questions (like "Does London have significantly slower journey times"), just establishing a ranking. So e.g. if I produce a ranking of journey times via train, is this more reliable than a ranking of journey times by car? $\endgroup$
    – Ollyver
    Commented Jun 19, 2017 at 19:14
  • $\begingroup$ stats.stackexchange.com/questions/642470/… $\endgroup$
    – Galen
    Commented Apr 11 at 15:43

1 Answer 1

Reset to default
2
$\begingroup$

For this purpose, you can use Kruskal-Wallis to test if there is any ranking that can be considered. https://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance

If it is the case, you can use Dunn's test for pair-wise comparisons.

These comparisons can form the ranking/ordering you expect.

If you assume your $y$ to be normal with the same variance in each group, then ANOVA and t-tests can be used.

Note that the ranking/ordering may not be linear as some pairwise comparison may exhibit differences that are not statistically significant, as e.g. Bob and Jane in the following example: http://www.quality-assurance-solutions.com/images/ANOVA-3.jpg

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ doyenne.com appears to now be a clothing website. $\endgroup$
    – Ollyver
    Commented Jun 20, 2017 at 6:04
  • 1
    $\begingroup$ @Ollyver Updated the link to dunntest. :) $\endgroup$
    – Alexis
    Commented Oct 11, 2017 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.