What is an appropriate Evaluation Metric and corresponding Loss function which best optimize the metric for a classification based FAQ Chatbot?

Question

I am developing a FAQ chatbot to display/return only one correct answer in a chat window for a given question from the user.

I know MRR & MAP make sense as an evaluation metric for information retrieval when you want to display multiple potential correct answers like in search engine. Unlike, a search engine, a chatbot should only return one correct. So I don't think MRR & MAP makes sense for chatbot.

Questions

1. What is an appropriate Evaluation Metric and corresponding Loss function which directly/best optimize the metric for a classification based FAQ Chatbot model?
2. Explain how the Loss function directly optimize the proposed Evaluation Metric?