6
$\begingroup$

I have the following problem:

The goal is, to find a model that classifies samples as risky, or less risky. However, only the risky samples are actually being manually investigated, i.e. labelled. Until now, a fixed model has been used for classifying samples into high risk and low risk, and then the ones deemed risky were further investigated, and it was concluded, if it was a false alert, and otherwise escalated.

That means, the existing labelled data, as well as any future labelled data, will be biased towards what was deemed risky by the fixed model, and a lot of true positives will be missed because of that.

Is there a smart way to decrease the impact of this selection bias when training a supervised model?

Improve this question
$\endgroup$
3
  • $\begingroup$ It's not entirely clear to me what data you're working with. I quote: "However, only the risky samples are actually being manually investigated, i.e. labelled. Until now, a fixed model has been used for classifying samples into high risk and low risk" . You write only the 'risky' samples are labeled - but what are the actual labels, 'risky' and 'not risky'? Or does 'risky' simply mean, that you're very uncertain about what the true label might be? $\endgroup$
    – deemel
    Commented Dec 14, 2017 at 13:47
  • $\begingroup$ Risky means that a legacy algorithm (which was bad, and needs to be replaced) has classified them as risky. I want to replace this algorithm with a better algorithm, but the legacy algorithm has resulted in a very biased training set $\endgroup$
    – Sam
    Commented Dec 15, 2017 at 8:24
  • $\begingroup$ Correct me if I am wrong but I think that you mean there is oversampling. I am not sure if this would help but IMHO, it is worth taking a look at it. communities.sas.com/t5/SAS-Data-Mining-and-Machine/… $\endgroup$
    – TheN
    Commented Dec 18, 2017 at 7:01

2 Answers 2

Reset to default
3
+50
$\begingroup$

1) You have no data on the false negatives (i.e. cases that are risky that the existing models deems 'not risky') which makes these cases impossible to identify.

2) You can train a new model that would distinguish between the true positives and false positives that you have data about i.e. the samples samples that have been manually investigated because the existing model deems them to be risky. However, this would not tell you anything about the data that have not been manually investigated. You would still not have the ability to correct for the false negatives in your data.

But this means that you could probably improve on the existing model by coupling the existing model with the new model. That is, use the two models sequentially, with the new model only being applied to the data that the existing model deems to be risky.

3) If your existing model is not a black box, you could attempt to learn its 'decision rules' and use that to develop a new model that functions alone (instead of two coupled models as suggested above). However, you still cannot improve performance on the false negatives without more data on this.

Improve this answer
$\endgroup$
1
$\begingroup$

Under certain assumptions, one way to capture some of those hidden false negatives would be to do clustering and possibly outlier detection. Then you can additionally to the high risk examples, manually examine some representative examples from each cluster, as well as / or the outliers. Also you can see if your clusters / model of the data significantly change over time.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ One practical application of such could be to use a tsne projection on the 1 before last layer of a neural network and using dbscan clustering. tsne projects higher dimensional samples to a lower dimension, and dbscan does not use a spherical approach like k-means. This way it is indeed possible to track if samples exhibit 'known' behavior, or are somehow different. It would need more evaluation initially, but indeed can end up with an efficient process to enhance the learning of the model. $\endgroup$
    – spdrnl
    Commented Dec 20, 2017 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.