I've built a (deep learning) model for a binary classification problem. For this problem I'm interested in sub-dividing my data set to maximize performance (as measured by precision) by essentially throwing out hard-to-predict samples. This would leave a smaller set of easier-to-predict samples with higher precision. How do I automate this?
For example say I'm predicting whether images are a hot dog or not (sorry). My overall performance as measured by precision is Y. If I pre-segment images into those with, say, key hot dog colors (red and yellow) and those without, my precision could be >Y for the images that pass the filter. I could set a lower score threshold (/ higher tolerance) for pass images and a higher score threshold (/ lower tolerance) for non-pass images. This would reduce the recall but boost the precision.
Is there a technique that automatically boosts precision by filtering in this manner? I.e. by varying score thresholds on different sub-populations? For my application, even reducing the population by 10x or 100x to a group of "sure thing" predictions would be beneficial.
Edit: Techniques like bagging or Adaboost might be the answer but AFAIK these methods don't explicitly try to remove samples but just adapt the model to sub-populations. Not sure if this is the same thing. Any insight?