1
$\begingroup$

Recall, Kendall distance is a manipulation of Kendall tau, in that it only considers discordant pairs, and therefore is a dissimilarity measure. It ranges from 0-1 and is calculated as follows:

$d= \frac{discordantPairs}{allPairs}$

Is there any significance test for this? How can the result of a comparison of two ranked lists be tested for significance? I am having trouble thinking of the right null hypothesis. -- One possibility is to say how likely is the observed d to occur if two lists of size n were permuted thousands of times. Does this approach makes sense with a distance metric?

Btw, here are the results from a 1000 run permutation of n=16, with the red lines indicating 0.05 and 0.95 significance levels respectively.

enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ "Is there any significance test for this?" - in relation to which hypothesis? (i.e. what statement about a population are you trying to test?) $\endgroup$
    – Glen_b
    Commented Mar 23, 2020 at 7:41
  • $\begingroup$ That's kind of what I'm confused about. What is the right null hypothesis for a distance-based measure? I guess, "that the distance observed between two ranked lists is not due to chance"? $\endgroup$
    – ha554an
    Commented Mar 23, 2020 at 20:18

1 Answer 1

Reset to default
0
$\begingroup$

There's certainly a test for it. The usual test for the Kendall correlation (that the population correlation is 0 vs the alternative that it's not 0) would correspond (under the usual assumption of continuity) to a test of the null that the normalized Kendall distance is $\frac12$ against the alternative that it differs from $\frac12$ (for the two-sided test; you can also do one sided tests).

Which is to say the p-value for one will be the p-value for the other.

It's unclear what hypothesis you had in mind though*, or whether this test would be any use to you.

With continuity (hence no ties), $n_c+n_d=n$, and $τ=(n_c−n_d)/n$ $=(n−n_d−n_d)/n$ $=1-2n_d/n$ $=1-2d_K$. So $τ=0⟹d_K=\frac12$.

* (you start with a hypothesis and then find a test -- if you don't have one, why would you contemplate testing anything?)

Improve this answer
$\endgroup$
10
  • $\begingroup$ So would I use the significance value from a Kendall-tau correlation? How did you arrive at 1/2? I can't seem to find any literature about how distance should be interpreted and if less or greater than 1/2 is some sort of a boundary. I'll tell you my situation, I am trying to compare 3 ranked lists and get an understanding of how far away each is from one another. $\endgroup$
    – ha554an
    Commented Mar 23, 2020 at 20:23
  • 1
    $\begingroup$ "how far away" is not a testing question. $\endgroup$
    – Glen_b
    Commented Mar 23, 2020 at 22:44
  • 1
    $\begingroup$ I have edited my answer $\endgroup$
    – Glen_b
    Commented Mar 23, 2020 at 22:54
  • 1
    $\begingroup$ By chance under what assumption? Testing against 1/2 - as already discussed - corresponds to testing whether they're uncorrelated (as Kendall measures it), but why would rejecting that null be interesting here? Surely the correlation is not actually 0, why would we consider otherwise? (Indeed, why is any hypothesis test interesting here?). You need to be clearer about what you need to find out. $\endgroup$
    – Glen_b
    Commented Mar 24, 2020 at 1:21
  • 1
    $\begingroup$ 1. "determine whether an individual's ranked preferences of n items changed from one-time point to another" -- you can see all the individual's ranked preferences at both time points and whether they were the same, so that appears not to be a subject for a hypothesis test (what's the population, and what's the hypothesis about it?). 2. "I meant if the two lists being compared were random permutations" -- what has that to do with the question about change in preferences over time? A small change would not be random but would presumably be of interest. Why random permutations? $\endgroup$
    – Glen_b
    Commented Mar 24, 2020 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.