I am developing a FAQ chatbot to display/return only one correct answer in a chat window for a given question from the user.
I know MRR & MAP make sense as an evaluation metric for information retrieval when you want to display multiple potential correct answers like in search engine. Unlike, a search engine, a chatbot should only return one correct. So I don't think MRR & MAP makes sense for chatbot.
Questions
- What is an appropriate Evaluation Metric and corresponding Loss function which directly/best optimize the metric for a classification based FAQ Chatbot model?
- Explain how the Loss function directly optimize the proposed Evaluation Metric?