# Overall rank from multiple ranked lists

I've looked through a lot of literature available online, including this forum without any luck and hoping someone can help a statistical issue I currently face:

I have 5 lists of of ranked data, each containing 10 items ranked from position 1 (best) to position 10 (worst). For sake of context, the 10 items in each lists are the same, but in different ranked orders as the technique used to decide their rank is different.

Example data:

List 1      List 2      List 3     ... etc
Item 1     Ranked 1    Ranked 2    Ranked 1
Item 2     Ranked 3    Ranked 1    Ranked 2
Item 3     Ranked 2    Ranked 3    Ranked 3
... etc

I am looking for a way to interpret and analyse the above data so that I get a final result showing the overall rank of each item based on each test and its position, e.g.

Result
Rank 1 = Item 1
Rank 2 = Item 3
Rank 3 = Item 4
... etc

So far I have attempted to interpret this information from performing Pearson's Correlation, Spearman's Correlation, Kendall Tau's B, and Friedman tests. I have found however, that these results have generally paired my lists (i.e. compared list 1 to list 2, then list 1 to list 3 .. etc), or have produced results such as Chi-Square, P-Values etc about the overall data.

Does anyone know how I can interpret this data in a statistically sound method (at a post graduate / PhD applicable level) so that I can understand the overall ranks signalling the importance of each item in the list across the 5 tests please? Or, if there is another type of technique or statistical test I can look into I would appreciate any hints or guidance.

(It maybe also worth noting, I have also performed the simpler mathematical techniques such as sums, averaging, minimum - maximum tests etc, but do not feel these are statistically important enough at this level).

Any help or advice would be greatly appreciated, thank you for your time.

– whuber
Apr 22 '13 at 12:58
• Thanks for your response. I've had a look at these articles, and i'm not sure whether they aren't what i'm looking for, or whether it's my understanding at fault. I get the impression in these articles that each of the data sets have many variables of different meanings, and that the ranks can be different or have more details integer values than just the rank. I am just looking for a statistically proven way to be able to say 'overall the most important item is item X, followed by Y ... and lastly (or least important) item Z'. I'm almost considering analysing these ranks 1-10 as plain numbers
– Liam
Apr 22 '13 at 13:37
• One major point of those threads is that there does not exist any such "statistically proven way." It is a question of valuation: any statistical combination of your results reflects a sense of tradeoffs among them. E.g., your "objects" might be cars and the "techniques" might rank them according to various attributes: cost, fuel efficiency, power, comfort, etc. Your personal sense of the "best" may differ substantially from some one else's sense and both of you would be right.
– whuber
Apr 22 '13 at 14:59
• did you get the answer? please leave comment here stats.stackexchange.com/questions/347336/… May 21 '18 at 10:25
• Perhaps, this is a good starting point: Combining ranking information from different sources in ranked-set samples onlinelibrary.wiley.com/doi/10.1002/cjs.11656?af=R Ranking and combining multiple predictors without labeled data ncbi.nlm.nih.gov/pmc/articles/PMC3910607 Dec 22 '21 at 12:44

I am not sure why you were looking at correlations and similar measures. There doesn't seem to be anything to correlate.

Instead, there are a number of options, none really better than the other, but depending on what you want:

Take the average rank and then rank the averages (but this treats the data as interval)

Take the median rank and then rank the medians (but this may result in ties)

Take the number of 1st place votes each item got, and rank them based on this

Take the number of last place votes and rank them (inversely, obviously) based on that.

Create some weighted combination of ranks, depending on what you think reasonable.

• An important point made in the threads I referenced in a comment--and I think this is the crux of the entire issue--is that all these methods are arbitrary. There exist objective methods but they require the use of information not inherent in the data. That's what makes this a problem of valuation rather than statistics.
– whuber
Apr 22 '13 at 14:56
• What weighted combination of ranks would you suggest? Dec 20 '16 at 13:22

As others have pointed out, there are a lot of options you might pursue. The method I recommend is based on average ranks, i.e., the first proposal of Peter.

In this case, the statistical importance of the final ranking can be examined by a two-step statistical test. This is a non-parametric procedure consisting of the Friedman test with a corresponding post-hoc test, the Nemenyi test. Both of them are based on average ranks. The purpose of the Friedman test is to reject the null hypothesis and conclude that there are some differences between the items. If it is so, we proceed with the Nemenyi test to find out which items actually differ. (We don't directly start with the post-hoc test in order to avoid significance found by chance.)

More details, such as the critical values for these both tests, can be found in the paper by Demsar.

Use Tau-x (where the "x" refers to "eXtended" Tau-b). Tau-x is the correlation equivalent of the Kemeny-Snell distance metric -- proven to be the unique distance metric between lists of ranked items that satisfies all the requirements of a distance metric. See chapter 2 of "Mathematical Models in the Social Sciences" by Kemeny and Snell, also "A New Rank Correlation Coefficient with Application to the Consensus Ranking Problem, Edward Emond, David Mason, Journal of Multi-Criteria Decision Analysis, 11:17-28 (2002).

I (well, Google) found a paper that benchmarks methods for combining ranked lists:

Li, X., Wang, X. and Xiao, G., 2019. A comparative study of rank aggregation methods for partial and top ranked lists in genomic applications. Briefings in bioinformatics, 20(1), pp.178-189. https://doi.org/10.1093/bib/bbx101

They use two R packages: TopKLists: https://cran.r-project.org/web/packages/TopKLists/index.html RobustRankAggreg: https://cran.r-project.org/web/packages/RobustRankAggreg/index.html

• Hi, welcome to CV. Please add the reference of the paper in case the link dies in the future. Feb 3 '21 at 12:48