0
$\begingroup$

I am writing a paper that is meant to establish that certain algorithms rank candidate values accross multiple problems in a different way. To do this, I take Algorithm A and run it on the candidate values and do the same for candidate B. When I run them, I notice that for multiple problems they rank the values as follows:

     A     B

c1   1     1
c2   2     1
c3   3     1

Is there a metric that can tell me something about how simillar these 2 ranking sets are without saying the correlation is undefined. fyi. I've tried Kendall's tau and it is undefined for the example given and so therefore I am looking for a metric that can tell me something useful in this particular case.

Improve this question
$\endgroup$
5
  • $\begingroup$ Do you have a true ranking to which you can compare or are you more concerned with how different the rankings are apart from a known measure of accuracy? $\endgroup$
    – Todd D
    Commented Sep 23, 2019 at 3:55
  • $\begingroup$ I am more concerned with how different the rankings are. $\endgroup$
    – Teererai Marange
    Commented Sep 23, 2019 at 4:59
  • $\begingroup$ Can your provide some pseudo-data. I am not sure I understand what a 0 column represents. Is Tau-b = 0 or all predictors zero for either or both of the candidate values? $\endgroup$
    – Todd D
    Commented Sep 23, 2019 at 22:05
  • $\begingroup$ Edited and added an example. A 0 variance column is one where all the candidates are tied eg. the B column on the example given. $\endgroup$
    – Teererai Marange
    Commented Sep 24, 2019 at 2:03
  • $\begingroup$ I don’t know that there is a fix for this scenario. Can you calculate the index using something other than whole integers? $\endgroup$
    – Todd D
    Commented Sep 25, 2019 at 13:56

1 Answer 1

Reset to default
1
$\begingroup$

I found a solution. It's a metric called Wilsons e by Wilson(1970). See Gonzalez et al. (1996) for details

Bibtex below: 

@article{gonzalez1996measuring,
  title={Measuring ordinal association in situations that contain tied scores.},
  author={Gonzalez, Richard and Nelson, Thomas O},
  journal={Psychological bulletin},
  volume={119},
  number={1},
  pages={159},
  year={1996},
  publisher={American Psychological Association}
}
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.