# Skewed Classification Problem

So I've read around and seen this is a problem. I have a classification problem and 12 variables ... I'm working on getting more, but even if l get the number to 20-30 I feel like the problem will still persist because predicting all 0s still gives like 99.99% accuracy.

The problem: I have 250 1's and 4,000,000 0's in my data set. I need to get those 250 1's correct from an algorithm (in a test set) and dont mind throwing in a few thousand 0's.

What I'm doing right now: Sampling 1000 of the 4,000,000 0's and using a binary classification boosting tree model using all 250 1's.

What I want to know: What are some of the things to consider if I were to train say 200 different models using the same 250 1's and doing a random sample of 1000 from the 4,000,000 each time and then aggregating the the final decision over the entirety of the model 'ensemble' if you will?

## 1 Answer

I think the following issues are worth keeping in mind:

• By training with this set sample ratio 250/1000 you are implying a specific cost function. This might or might not be the cost function you want.
• The process you're outlining is commonly done with the randomForest package: stratified or internal undersampling.
• You are still at risk of overfitting. (1) This is a small absolute number of positive outcomes beyond the issue of the ratio of outcomes. This is a more challenging issue in terms of model creatiion (2) How do you obtain accurate OOB estimates with such a small sample of positives?
• You should decide on how to measure model performance. Accuracy is likely not what you want.
• Thank you. I'm going to try segment my over sided population further to cut out non overlapping segments to try reduce the skewed proportions then try again.. – koala_man_fan Mar 30 '15 at 15:24