Evaluation of machine learning model in production

Question

I know this question would require the knowledge of environment that the model is deployed in. So, I'm going to disclose as much knowledge as possible while being discreet.

The actual question -
I developed a machine learning model for document classification. It was trained offline using batch training i.e. (trained and tested once using a fixed data set). The model gives a single label out of the several labels seen in the data set.

This model is going to be used by the clients who are humans. The model classifies the document that they are reading and shows the prediction to them. The human/user is allowed to correct the prediction if its wrong. My idea was to use this correction as a feedback to my model ( How the feedback is used is explained below )

But the problem is that the user/human is not actively correcting the tag i.e. he/she doesn't correct it if its wrong ( because there is no incentive ). I was using #corrections to the displayed prediction, as an evaluation measure i.e. greater #corrections within a specific time period implies poor performance and vice-versa.

Coming back to the way feedback is fed to the model, my idea is use a correction to generate a feedback data set. And next ( when the size of feedback data set is significant ), I train the model again using the feedback data set i.e. flash the data set to the model for a number of epochs ( the model is capable of incremental learning )

My questions -

1. How do I evaluate this model ? Assuming that the human/user doesn't correct the prediction, are there any other evaluation metrics ? (The only thing that a human/user sees is the document that he/she is reading and the prediction given by the model.) Are there any other implicit ways to get the user to engage in improving the model ? Are there any other metrics that don't involve comparing predictions and ground truth ?

2. Assume that the 1st problem is solved i.e. I can get the human/user to correct the prediction. How often do I train the model ? What should be the minimum size of the feedback data set ?

I know this sounds like a broad question. But, there are a few aspects that are closely related to Machine learning. Please help with what you feel is relevant here else point out the appropriate resources / help me migrate this question to appropriate community.

Answer 1

The first question on “how to make humans give feedback” seems more like a UI question. I won’t answer it here.

The second question seems like active machine learning. This is where the model learns in an “online” fashion, and continually updates. There’s an entire literature on this.

If that’s what you want, there are some tools: prodigy https://prodi.gy/ and NEXT http://nextml.org/. Prodigy seems best suited for NLP, as it’s developed by the same people as spacy.