1
$\begingroup$

I am developing a FAQ chatbot to display/return only one correct answer in a chat window for a given question from the user.

I know MRR & MAP make sense as an evaluation metric for information retrieval when you want to display multiple potential correct answers like in search engine. Unlike, a search engine, a chatbot should only return one correct. So I don't think MRR & MAP makes sense for chatbot.

Questions

  1. What is an appropriate Evaluation Metric and corresponding Loss function which directly/best optimize the metric for a classification based FAQ Chatbot model?
  2. Explain how the Loss function directly optimize the proposed Evaluation Metric?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.