Suppose I have trained an image classifier that predicts the prominent objects in an image and the output predictions of this model is displayed to end users of some application.
After deployment (i.e at inference time) I would like to display the classifier’s output predictions only if they are “reasonably correct”. Let’s say showing obviously wrong predictions would reduce user engagement with the application.
One way to do tackle this issue could be to train a secondary classifier that takes the image and my model’s predictions as input and make it generate a binary display / don’t display label. Let’s say using the first classifier’s output probability is not very reliable because it can be wrong with high confidence/probability.
Let’s assume I can get some training data for the second classifier as well. To get a reasonable secondary classifier let’s say I train it on a balanced dataset comprising of instances where the first model does well 50% of the time (display label)and when the first model does badly the remaining 50% (don’t display).
My questions is — is there an inherent error in this setup? I’m using a secondary ML model to decide if my original ML models output is good or not? It feels incorrect but I can’t find any literature on why this is not ok. Can anyone justify why this is either ok or not? Under what conditions would this be ok to do? Any references would be great too.