7
$\begingroup$

I am doing data analysis in the mobile ad targeting domain. I have around 18 features and for a combination of these features, the result is either True or False (1/0) depending on whether the impression was clicked or not. The problem here is that the output class is highly skewed. Click though rate is around 0.4%. (i.e value is 1 only 4 out 1000 times). I have a data set of 2 million rows and I am using 90% as train set and 10% as test set. I have used logistic regression from sckit-learn package in python. Now after training my model I get all values for test set as 0. Please tell me what the problem could be and what should I do to solve it?

P.S. : I have tried increasing my data set size and also reducing the number of features(even to just one feature). If I see the probability of each class(0/1) in the test set, I get around 0.002 - 0.005 for a 1.

Thanks

Improve this question
$\endgroup$

3 Answers 3

Reset to default
14
$\begingroup$

First 18 isn't a lot of features at all and you should see if you can get more data. Google uses a ridiculous number of features in their ad targeting and takes a different online/game theoretical approach to choosing what ad to show to the audience

Second, skewed class labels like this are a common problem. Search terms to look at include imbalanced or unbalanced classification and "skew insensitive". There are a bunch of approaches you can and should try:

  • Stratified cross validation to make sure you end up with enough positives in the test.
  • Under/over sampling as others have mentioned or roughly balanced bagging for random forests. There are also methods for generating new minority class samples and sampling representative majority class samples. I saw a python library for this here.
  • Class weighted or cost sensitive learning can work well and there are versions of many methods that can do this (though not in scikit learn).
  • Boosting (gradient or adaptive) can work well.
  • Transductive or one class approaches which treat the data as positive and unlabeled can work well though they assume the positives are members of a larger class of possible positives.
  • Hellinger distance decision trees are gaining some bus for working well on unbalanced data.

Most of these approaches essentially reflect that you care more about getting the positives right then getting the negatives wrong. Within scikit.learn you're limited in the number of these you can try without some custom code but there are lots of other libraries out there if you google around though they'll be in a mix of languages.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ "Class weighted or cost sensitive learning can work well and there are versions of many methods that can do this (though not in scikit learn)." – scikit-learn's SVMs, SGDClassifier, and its trees/forests all support class_weight parameters. Maybe some others too. $\endgroup$
    – Danica
    Commented May 24, 2015 at 19:38
5
$\begingroup$

I'll try to add a little intuition as to why you get such results. Considering all such classification tasks, the best result would be to predict 100% of the results correctly, right? In your case, with 0.4% vs 99.6% class balance, if you predict 0 for every row, then you automatically get to be 99.6% right. That's is a very very good result!

As for how to approach these problems - as far as I know, there is no algorithms that work with very skewed classes. Hence there's two ways to approach it, just as DSea described, one is oversampling and the other is undersampling.

In case of oversampling you add the smaller class many times. If you start out, as you do, with 1:250 ratio of classes, you might want to take the smaller class 50 times, so you end up with 50:250 or 1:5 ratio, which should already work with most classification algorithms. You'll have to keep in mind of course that each sample of the positive class is 50 times more "important" now.

In case of undersampling you'll aim for a similar ratio, but achieve it by just picking 5 random samples from the larger class for every one of the smaller class. The drawback here is that you're looking only at a tiny part of the whole dataset.

So there are ways to work with the data you have, but everything is a bit more complicated than it seems in the beginning :)

Improve this answer
$\endgroup$
1
  • $\begingroup$ That 99.6% accuracy is exposed as rather poor when it is presented in the proper context. This relates to something I read in an elite economics journal where a model was implied to be very good due to its high classification accuracy of 97% when 97.71% of the outcomes belonged to the same category, discussed in my answer here. (The article is interesting and is a useful contribution to financial economics, but they do make that mistake.) $\endgroup$
    – Dave
    Commented Jul 5 at 9:36
2
$\begingroup$

The problem is the skew of the class balance. The simplest thing you could try would be to reduce the size of the majority class of your training set. Just randomly sample (without replacement) N instances form the majority class, where N is the number of instances in the minority class. This is called 'undersampling.'

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.