1
$\begingroup$

So I've read around and seen this is a problem. I have a classification problem and 12 variables ... I'm working on getting more, but even if l get the number to 20-30 I feel like the problem will still persist because predicting all 0s still gives like 99.99% accuracy.

The problem: I have 250 1's and 4,000,000 0's in my data set. I need to get those 250 1's correct from an algorithm (in a test set) and dont mind throwing in a few thousand 0's.

What I'm doing right now: Sampling 1000 of the 4,000,000 0's and using a binary classification boosting tree model using all 250 1's.

What I want to know: What are some of the things to consider if I were to train say 200 different models using the same 250 1's and doing a random sample of 1000 from the 4,000,000 each time and then aggregating the the final decision over the entirety of the model 'ensemble' if you will?

$\endgroup$
0
$\begingroup$

I think the following issues are worth keeping in mind:

  • By training with this set sample ratio 250/1000 you are implying a specific cost function. This might or might not be the cost function you want.
  • The process you're outlining is commonly done with the randomForest package: stratified or internal undersampling.
  • You are still at risk of overfitting. (1) This is a small absolute number of positive outcomes beyond the issue of the ratio of outcomes. This is a more challenging issue in terms of model creatiion (2) How do you obtain accurate OOB estimates with such a small sample of positives?
  • You should decide on how to measure model performance. Accuracy is likely not what you want.
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you. I'm going to try segment my over sided population further to cut out non overlapping segments to try reduce the skewed proportions then try again.. $\endgroup$ – koala_man_fan Mar 30 '15 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.