2
$\begingroup$

To make my problem more clear I will give some examples of my data:

Measure A Measure B
Set 1 5 3 1 4 2 5 4 2 3 1
Set 2 5 1 3 4 2 5 4 1 3 2
Set 3 2 4 3 5 1 5 2 3 4 1
Set 4 3 4 1 5 2 5 3 2 4 1
Set 5 5 3 2 4 1 5 4 2 3 1

I have multiple sets of items to be ranked and multiple measures by which the items in a set can be ranked. Each cell in the table presents a ranking of the items in one set by one of the measures (e.g. rankings of the 5 items in set 1 as ranked by measure A are in the cell in at line 2 column 2 of the 3x3 table).

My goal is to check if Measure A ranks items differently than Measure B. I want to check if there is a statistically significant difference between the way that the two measures ranked the items in the different sets.

While searching I only found ways to compare 2 rankings between them (e.g. comparing the ranking of measure A on set 1 and the ranking of measure B on Set 1), but not ways to compare multiple rankings lists.

Improve this question
$\endgroup$
4
  • 4
    $\begingroup$ I am not clear what you would count as an insignificant difference. You can see they do produce different rankings, as all of you sets show differences and you only needed one such set to say this $\endgroup$
    – Henry
    Commented Dec 5, 2021 at 16:41
  • 1
    $\begingroup$ What it sounds like to me is that you'd like to construct a distance metric on $S_n$. This means a quantitative way of computing "how dfiferent" two permutations of $n$ items are. I recommend looking into a few different distance metrics on $S_n$ to find which one best fits your application. EDIT: Here's a helpful survey of some metrics: cicirello.org/publications/cicirello-bict2019.pdf $\endgroup$
    – dsillman2000
    Commented Dec 5, 2021 at 16:57
  • $\begingroup$ I am sorry, english is not my first language and maybe I did not explain correctly. I want to try and prove that Measure A ranks things differently than Measure B. I can't just say that they are kind of different sometimes. I have 300 sets of items that can be ranked, and later there will be even more. I took a sample of 40 sets to analyuze and I want to use that sample to show that, in general Measure A is different form Measure B, not coincidentally sometimes different. $\endgroup$
    – Andrei Mihai
    Commented Dec 6, 2021 at 10:31
  • $\begingroup$ Like when you measure the blood pressure of patients before and after treatment you want to prove that the mean indeed changed significantly, and you need the paired data t-test to prove that, you can't just say that they seem different. I was asking what kind of test or design I could use to prove that statistically, with a p-value and so on. $\endgroup$
    – Andrei Mihai
    Commented Dec 6, 2021 at 10:31

1 Answer 1

Reset to default
1
$\begingroup$

Your problem might be attacked by performing a permutations test on the ranks for each of the items in the sets (i.e. those labelled '1' through '5' in your table) to determine how strongly the data cast doubt on the hypothesis that the rankings differ between the two measures (methods?) only randomly.

Take each item and tabulate their ranks according to the two methods measures. (Using numbers to represent the items in your table is here not entirely helpful, so maybe use a,b,c... instead.) The ranks for item '5' in the table are 1,1,4,4,1 for measure A and 1,1,1,1,1 for measure B. The sum of the differences in ranks would be a useable statistic here, so a summary if the differences between rankings is 6 (i.e. (1-1)+(1-1)+(4-1)+(4-1)+(1-1)=6). That statistic treats the rankings as paired across measures.

Is 6 large or small? See where it lies in the distribution of all possible re-arrangements of the ranks across the measures by permutations.

Improve this answer
$\endgroup$
1
  • $\begingroup$ +1 for the paired permutation test $\endgroup$
    – Ggjj11
    Commented Nov 29, 2023 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.