I have the following problem:
The goal is, to find a model that classifies samples as risky, or less risky. However, only the risky samples are actually being manually investigated, i.e. labelled. Until now, a fixed model has been used for classifying samples into high risk and low risk, and then the ones deemed risky were further investigated, and it was concluded, if it was a false alert, and otherwise escalated.
That means, the existing labelled data, as well as any future labelled data, will be biased towards what was deemed risky by the fixed model, and a lot of true positives will be missed because of that.
Is there a smart way to decrease the impact of this selection bias when training a supervised model?