How can I calculate inter-rater agreement for two raters assigning ranks?

Question

I am new to statistics and currently trying to calculate the inter rater agreement between two entities based on their rankings of a set of things. For example if i give person 1 a set of 5 things and ask him to rank them based on order of preference and give the same set to person 2 and ask him to rank what he prefers how would i calculate their inter rater agreement?

I read about the ways to calculate and i guess the closest method it would resemble are methods used for Likert scales(i may be way off here!) but i am not sure it exactly fits my problem here. Can anybody help me with this?

Thanks!

Answer 1

You can use Kendall's tau for two raters or Kendall's W for more than two raters. Due to its intuitive appeal, I will include the formula for the tau coefficient below. Pairs of observations are said to be concordant if the ranks for both elements agree and are said to be discordant otherwise.

$$ \tau = \frac{(\text{number of concordant pairs}) - (\text{number of discordant pairs})}{n(n-1)/2} $$