4
$\begingroup$

I am developing a random forest model for predicting fraudulent credit card transactions. I have made a train and test split in my dataset, and finally chosen a model through different metrics, including accuracy, recall, and AUC. Before, I have had issues because of extremely unbalanced datasets (only 2% fraudulent transactions). After some oversampling, I have used a 50%/50% fraud/no fraud dataset to train my model. Now, this model, and its fitted classifier, will be used in production mode. Is it legitimate to use this classifier, trained with a balanced dataset, even if the transactions that it will be classifying will be mostly not fraudulent? Won't it be biased towards classifying transactions as fraudulent?

EDIT:

In order to evaluate my model (implemented in scikit-learn) I was using the scores obtained from the train test split provided as a built-in method. I realized this might be providing optimistic accuracy, recall, and AUC scores, and it was probably overfitting the mode. So, I decided to use scikit-learn k-fold cross validation. The results obtained through this method are much worse. For example, recall used to be 69% when evaluating the model against the test data, but it is 18% when using 5 fold cross validation (mean of recall scores per iteration). This improves a little if I modify the class_weight parameter to {0:0.99, 1:0.01}, but this doesn't makes sense, I think, as it penalizes errors in the classification of 0's as 0's and not the other way around, that is, for the more uncommon events (1's or positives). Does this mean my model is overfitting? Which measure is more accurate to evaluate the real world performance of my model? Does it even make sense to use cross-validation with random forests?

$\endgroup$
  • $\begingroup$ Replying to you edit: when estimating the performance you should consider two points. One is that your estimation is not biased and the other is that your validation set is representable. Using the test set is indeed too optimistic. Your CV results are reliable. However, remember that your balanced dataset is not representable. I think that estimating on the imbalanced dataset is more reliable (and easier) than working with weights. $\endgroup$ – DaL Jun 23 '16 at 6:32
1
$\begingroup$

This is actually an interesting question that comes across alot in medical data. One of the ways to understand oversampling and classification of unbalanced data is because oversampling is an active bias of sampling the data, the results will be biased. When compensating for the minority class, remember that the goal of classification is to identify characteristics that can determine which class an outcome can belong to and then address how the independent variables interact.

When oversampling data for classification, remember to use cross-validation properly and to oversample data during the cross-validation as opposed to before the cross-validation. This will give you better (more accurate) scores with sensitivity and specificity and limit (though not eliminate) the effect of bias and overfitting due to improperly using cross-validation and oversampling.

Here is a good reference using preterm births: http://www.marcoaltini.com/blog/dealing-with-imbalanced-data-undersampling-oversampling-and-proper-cross-validation

$\endgroup$
1
$\begingroup$

If your classifier works well at production mode, it is legitimate to use it. You can build it in whatever method you like.

Working on a balanced dataset is a good way to build a model that differ between the majority and minority classes. However, as @akash87 and you noted, it might cause a bias.

You might have been lucky and though the bias your model will perform well on the production data. In order to know that evaluate it on the original dataset too. For the usage of different datasets in order to learn and validate see here

In the more common scenario, the bias hurts the performance and you should adapt the model back to the production distribution. You can adapt your model back to the production distribution by learning a new model that will do this adaptation. For details see here.

You might be interested in this Editorial: Special Issue on Learning from Imbalanced Data Sets and Learning from Imbalanced Data

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.