Training models on biased samples I have the following problem:
The goal is, to find a model that classifies samples as risky, or less risky. However, only the risky samples are actually being manually investigated, i.e. labelled. Until now, a fixed model has been used for classifying samples into high risk and low risk, and then the ones deemed risky were further investigated, and it was concluded, if it was a false alert, and otherwise escalated. 
That means, the existing labelled data, as well as any future labelled data, will be biased towards what was deemed risky by the fixed model, and a lot of true positives will be missed because of that.
Is there a smart way to decrease the impact of this selection bias when training a supervised model?
 A: 1) You have no data on the false negatives (i.e. cases that are risky that the existing models deems 'not risky') which makes these cases impossible to identify.
2) You can train a new model that would distinguish between the true positives and false positives that you have data about i.e. the samples samples that have been manually investigated because the existing model deems them to be risky. However, this would not tell you anything about the data that have not been manually investigated. You would still not have the ability to correct for the false negatives in your data. 
But this means that you could probably improve on the existing model by coupling the existing model with the new model. That is, use the two models sequentially, with the new model only being applied to the data that the existing model deems to be risky. 
3) If your existing model is not a black box, you could attempt to learn its 'decision rules' and use that to develop a new model that functions alone (instead of two coupled models as suggested above). However, you still cannot improve performance on the false negatives without more data on this. 
A: Under certain assumptions, one way to capture some of those hidden false negatives would be to do clustering and possibly outlier detection. Then you can additionally to the high risk examples, manually examine some representative examples from each cluster, as well as / or the outliers. Also you can see if your clusters / model of the data significantly change over time. 
