Using a secondary classifier to evaluate a primary classifier?

Question

Suppose I have trained an image classifier that predicts the prominent objects in an image and the output predictions of this model is displayed to end users of some application.

After deployment (i.e at inference time) I would like to display the classifier’s output predictions only if they are “reasonably correct”. Let’s say showing obviously wrong predictions would reduce user engagement with the application.

One way to do tackle this issue could be to train a secondary classifier that takes the image and my model’s predictions as input and make it generate a binary display / don’t display label. Let’s say using the first classifier’s output probability is not very reliable because it can be wrong with high confidence/probability.

Let’s assume I can get some training data for the second classifier as well. To get a reasonable secondary classifier let’s say I train it on a balanced dataset comprising of instances where the first model does well 50% of the time (display label)and when the first model does badly the remaining 50% (don’t display).

My questions is — is there an inherent error in this setup? I’m using a secondary ML model to decide if my original ML models output is good or not? It feels incorrect but I can’t find any literature on why this is not ok. Can anyone justify why this is either ok or not? Under what conditions would this be ok to do? Any references would be great too.

Answer 1

One potential weakness is the following: if it is feasible to use some features to learn that your initial classifier will fail on some instances, then why don't you just use these exact features to improve the initial classifier in the first place?

And if you do this and iterate the process, then you should at some point be left with situations where you can't learn that your classifier will perform badly for a given instance (otherwise you would do another round of improving your classifier).

So yes, you can do this in principle, but it would be good to have a convincing answer if your users ask you why you don't just use the features in the second classifier to improve the first one. (I personally don't quite see such a good answer.)

Alternatively, this is related to boosting: train a weak classifier, and then train another one with a higher weight on instances that the first classifier got wrong.