0
$\begingroup$

I am trying to find the best "ranking metric" for my problem but could not find any existing suitable metric, can someone help.

I'm interested in information retrieval.

For item there is a "revelancy" between 0 and 1 that acts as a label. My algorithm gives a score to each item, and returns the 5 items with the highest score. I want to know how "good" my algorithm is to retrieve the real top 5 events, which are the 5 items with highest relevancy scores. Here are a few properties I'm interested in for my evaluation metric:

  • for items not selected (not in top 5), I don't care about their score
  • for items selected in top 5, I don't care about their order (ie returning A,B,C,D,E is the same as returning A,B,C,E,D)
  • I want that returning higher-ranked items is better than returning lower-ranking items. Ie the best result is to return the 5 items with highest relevancy score, and lowest result is to return the 5 items with lowest relevancy score.
  • I want to take into account that items with "close enough" relevancy score change the metric by not much. For instance, if I include in my result the item ranked 6th in relevancy instead of 5th in relevancy, I want this impact to be big is the relevancy score of 6 is way lower to 5, but almost nothing if the relevancy score of 6 and 5 are close.

Are their "famous" metrics that do this? I couldn't find any.

A criteria I was thinking about is to define my metric as : sum of relevancy of the 5 selected items / sum of relevancy of the 5 most relevant items.

This is somewhat doing what I want but it feels like I'm re-inventing an existing metric

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.