I am developing a FAQ chatbot to display/return only one correct answer in a chat window for a given question from the user.

I know MRR & MAP make sense as an evaluation metric for information retrieval when you want to display multiple potential correct answers like in search engine. Unlike, a search engine, a chatbot should only return one correct. So I don't think MRR & MAP makes sense for chatbot.


  1. What is an appropriate Evaluation Metric and corresponding Loss function which directly/best optimize the metric for a classification based FAQ Chatbot model?
  2. Explain how the Loss function directly optimize the proposed Evaluation Metric?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.