I am trying to find the best "ranking metric" for my problem but could not find any existing suitable metric, can someone help.
I'm interested in information retrieval.
For item there is a "revelancy" between 0 and 1 that acts as a label. My algorithm gives a score to each item, and returns the 5 items with the highest score. I want to know how "good" my algorithm is to retrieve the real top 5 events, which are the 5 items with highest relevancy scores. Here are a few properties I'm interested in for my evaluation metric:
- for items not selected (not in top 5), I don't care about their score
- for items selected in top 5, I don't care about their order (ie returning A,B,C,D,E is the same as returning A,B,C,E,D)
- I want that returning higher-ranked items is better than returning lower-ranking items. Ie the best result is to return the 5 items with highest relevancy score, and lowest result is to return the 5 items with lowest relevancy score.
- I want to take into account that items with "close enough" relevancy score change the metric by not much. For instance, if I include in my result the item ranked 6th in relevancy instead of 5th in relevancy, I want this impact to be big is the relevancy score of 6 is way lower to 5, but almost nothing if the relevancy score of 6 and 5 are close.
Are their "famous" metrics that do this? I couldn't find any.
A criteria I was thinking about is to define my metric as : sum of relevancy of the 5 selected items / sum of relevancy of the 5 most relevant items.
This is somewhat doing what I want but it feels like I'm re-inventing an existing metric