Correcting sample selection bias of binary classifiers

Question

In fraud investigation the number of detected fraud cases can be very small when compared to the total number of cases. This would also apply to rare desease detected in a very small number of people in the population. In these cases we have a limited number of investigated cases and from these cases we know the target (fraud cases or patients with the desease) and non-target (not fraud or patients without the desease) cases.

Since in these cases the target observations only account for, let say, 1% of the total number of cases we will have a very imbalanced data and most classification models will classify as non-target almost all the time, so it will not be very useful. One way to tackle this kind of problem is to use only the investigated cases in the training dataset so that we will have a less imbalanced data. The problem with this approach is that the distribution of the data used to build the model will be different from the data that it will make predictions on. In this case we deliberately have a selection bias since we are just using investigated cases, however there are ways to address this issue and obtain better results.

I have been searching for a technique to deal with this problem but there are few studies available. The best one I found so far is the following: On Selection Bias with Imbalanced Classes. This paper says that we should use a subsample of the unlabled data, that is the cases that were not investigated, and consider them as non-targets. However, in the paper I could not find exactly how to subsample the unlabled data (how many, which ones). I would like to know if anyone knows the steps to take in this kind of situation or any other techniques to address this issue.

Answer 1

When facing imbalanced problems there are a number of approaches: resampling (as you already pointed out), reweighting the samples according to the ratio of class samples, adapting/developing an algorithm to be robust against imbalance or using the ROC curve to find the optimal threshold.

There are many papers presenting some sampling schemas. One popular one is SMOTE (Synthetic Minority Over-sampling Technique). There are also implementations available to try out. In that paper you see references to previous works and problems with the different approaches. SMOTE makes some strong assumptions when generating data, and, as reported in the paper, there are cases where it actually worsens the performance, but in other cases helps quite a bit.

What they show is: one needs a better strategy to oversample the minority class (just sampling with replacement does not help). If combined with undersampling of the majority class, results can be further improved.

See also this other question in our forum for an example of how to make logistic regression perform well under this setting. There are a number of papers on how to adapt kNNs to cope with imbalance data sets (for example Class Confidence Weighted kNN Algorithms for Imbalanced Data Sets), and there are specially designed algorithms for particular tasks like novely detection (Support Vector Method for Novelty Detection).

There is a wealth of literature on the topic of imbalanced data sets (novelty/anomaly/fraud detection). But I hope these few couple of references help you to get started.

Answer 2

I know this is late, but if I understand your question correctly, I may have worked on a relevant solution to this. If by "investigated cases" you mean cases that were more likely to be in the target class than the general data population, then this approach should apply.

Generally, you can account for this selection bias by using importance-weighting of examples based on whether they are in the selection set or not. Note that this is different than reweighting cases based on their label, which is common in class-imbalance adjustment.

I will illustrate using the specific problem I worked on. Full details in Discovering Foodborne Illness in Online Restaurant Reviews.

In this case, we had a simple model that used keywords to classify Yelp reviews as either talking about food poisoning or not. (The target "positive" class was extremely rare, <1%, hence the use case for a model to find these rare examples.)

Labelers (domain experts) gave feedback on only the positive predictions of the model, so we eventually had a large training set, but it suffered from serious selection bias and was not representative of the true data population.

To fix this, we quantified this selection bias -- our model had partitioned all of the data into two sets, and so we could compute

p(selected by model) = # model positive predictions / # cases considered by model

p(not selected by model) = 1-p(selected by model)

To retrain an unbiased model using all the data, we took examples from both selected by the model and not selected by the model sets and reweighed them using inverse propensity weights. Our loss function was then rescaled by these weights depending on the example.

$$L(\theta;x_i,y_i) = \begin{cases} \frac{1}{p(\text{selected by model})} \ell(f_\theta(x_i), y_i) & \text{if } x_i \text{ predicted positive by model}\\ \frac{1}{p(\text{not selected by model})} \ell(f_\theta(x_i), y_i) & \text{otherwise} \end{cases} $$

Likewise, we computed bias-adjusted test metrics. It's analogous to the loss, where each individual example truth-prediction comparison is given the inverse propensity weight instead of a weight of 1. What this does essentially is count errors on the not selected by model subset as significantly more egregious, since they are underrepresented by in the evaluation data compared to the true data. This reflects the fact that one error on this small sample would mean hundreds of errors on the true data.

Hope this helps!