3
$\begingroup$

Say I have a set of my top 100 favorite movies from a list of 1000 possible choices

$$ \{M_1,M_2,..., M_{100}\} $$

And have asked my friends to guess my top movies by ranking them in order of confidence.

Each friend submits their ranking of movies,

$$ [F_{1,1},F_{1,2}, ..., F_{1,100 }] $$

I want to evaluate how well my friends know me by comparing their lists and seeing who ranked movies that I included the highest on average.

To do this, the naive things to do would be to take those movies from my friend's list that intersect with my movie set, and compute the average ranking over that list, and compare those rankings.

However, there is a correlation between the size of the overlap and the average ranking. That is, the more movies my friend correctly included in their list (of any rank) the higher their average ranking will be.

What is the appropriate transformation to decouple average rank from length of overlap, in this case?

Improve this question
$\endgroup$
4
  • $\begingroup$ Yes, average ranking is not a good statistic to consider. But it seems strange to me that you want to construct a better statistic by transforming the average ranking. Why not just make a new statistic from scratch? $\endgroup$
    – svendvn
    Commented May 7, 2019 at 23:46
  • $\begingroup$ Which statistic would you consider here, then @svendvn, that would capture the ability of a particular individual to rank highly movies that I chose in my list? $\endgroup$
    – Danny W.
    Commented May 8, 2019 at 0:05
  • $\begingroup$ NDCG (Normalised Discounted Cum. Gain) would be a good start. Mean Reciprocal Rank and Fraction of Concordant Pairs are also popular metrics. In general, I would re-cast this as an Information Retrieval task. $\endgroup$
    – usεr11852
    Commented May 9, 2019 at 0:28
  • $\begingroup$ I ended up using the AUROC and the AUPRC. They aren't perfect but they are understandable.Thanks everyone for your help. $\endgroup$
    – Danny W.
    Commented May 10, 2019 at 14:44

1 Answer 1

Reset to default
3
$\begingroup$

Let $M=\{M_1, \dots, M_{100}\}$ be the unordered set of movies that are your favorites. Let $\mathcal F=(F_1, \dots, F_{100})$ be the ordered list of movies that a friend selected. I assume that $F_{100}$ is the highest ranked movie. You have considered the overlap

$$ \text{overlap}=\sum_{i=1}^{100}\mathbb{1}(F_i\in M) $$ and the average ranking $$ \text{av. ranking}=\frac{1}{\text{overlap}}\sum_{i=1}^{100}i\cdot \mathbb{1}(F_i\in M) $$ The problem with overlap is that it does not take the ordering of the friend's movies into account and the problem with average ranking is that it penalizes large overlaps. I propose the total ranking $$ \text{total ranking}=\sum_{i=1}^{100}i\cdot \mathbb{1}(F_i\in M) $$ It takes ordering into account and it rewards large overlaps. It could also be done more generally $$ \text{generalized total ranking}=\sum_{i=1}^{100}g(i)\cdot \mathbb{1}(F_i\in M) $$ for example if you think that 100 is a bit arbitrary you could say $g(i)=900+i$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.