10
$\begingroup$

I am trying to understand when it is appropriate to use the MAP and when MRR should be used. I found this presentation that states that MRR is best utilised when the number of relevant results is less than 5 and best when it is 1. In other cases MAP is appropriate. I have two questions:

  • I don't really understand why this is so.
  • I can't find a citable reference for this claim.

Please note that I don't have a very strong statistical background so a layman's explanation would help a lot. Thank you.

$\endgroup$
17
$\begingroup$

Imagine you have some kind of query, and your retrieval system has returned you a ranked list of the top-20 items it thinks most relevant to your query. Now also imagine that there is a ground-truth to this, that in truth we can say for each of those 20 that "yes" it is a relevant answer or "no" it isn't.

Mean reciprocal rank (MRR) gives you a general measure of quality in these situations, but MRR only cares about the single highest-ranked relevant item. If your system returns a relevant item in the third-highest spot, that's what MRR cares about. It doesn't care if the other relevant items (assuming there are any) are ranked number 4 or number 20.

Therefore, MRR is appropriate to judge a system where either (a) there's only one relevant result, or (b) in your use-case you only really care about the highest-ranked one. This might be true in some web-search scenarios, for example, where the user just wants to find one thing to click on, they don't need any more. (Though is that typically true, or would you be more happy with a web search that returned ten pretty good answers, and you could make your own judgment about which of those to click on...?)

Mean average precision (MAP) considers whether all of the relevant items tend to get ranked highly. So in the top-20 example, it doesn't only care if there's a relevant answer up at number 3, it also cares whether all the "yes" items in that list are bunched up towards the top.

There's no real need to use MAP if you only ever have 1 relevant answer in your data, MRR would be fine. But if you submit a query such as "female heads of state" and the top three results are "Margaret Thatcher", "Vigdís Finnbogadóttir", and "Pratibha Patil", those all are distinct but correct answers, and we might want to judge this as a good result, better than a system which only floated one of those answers to the top.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.