I have been trying to learn a decision tree on a data-set with almost 400 features. The target variable has only two values and the data is highly skewed towards the non-event class (90 % of the data set).

All the features take only Boolean values. The decision tree plots are skewed towards the non event class. I am looking for the path that helps me predict the event class. I couldn't find a definitive answer anywhere. Can anyone give me some ideas?


Apologies if the question wasn't clear. I just started as a Data Science inter, and this is my first project.

The decision tree predicts almost all the non-event class (90% of the data) with high accuracy, whereas almost entirely misclassifies the event class.

What do I need to do to build the tree that predicts the non-event class with high accuracy and to see the decision paths that makes up such a tree?

I tried ensemble methods but it only improved my classifier marginally

  • $\begingroup$ Decision trees will work just fine. Try boosting or bagging to deal with the unbalanced classes. The model you train will output a score for each class; try different thresholds on these scores to obtain reasonable sensitivity and specificity values. $\endgroup$ Feb 25 '18 at 13:07
  • $\begingroup$ Why is the fact that the decision tree skews in the same way as the data a problem? It ought to do so! If it didn't it would be making a lot of wrong classifications. $\endgroup$
    – Peter Flom
    Feb 25 '18 at 13:27
  • $\begingroup$ Try to select the model parameters in a way that is dependent on the metric that is sensitive even to small predicted label changes. You might want to take a look at optimization of area under the curve (AUC) for ROC curve. $\endgroup$ Feb 26 '18 at 23:46

Clarification of the problem

To clarify the OP's problem, as I read it (and your edit makes this more clear): Given a dataset with 10% events and 90% nonevents, a naive classifier could obtain 90% accuracy simply by choosing the 'nonevent' outcome every single iteration.

Tree-based algorithms (as well as other types of classifiers) can easily fall into this minimization 'mistake' because the algorithm is designed to treat all errors as equal; in many real-world situations, this is simply not acceptable, and one may prefer an algorithm that makes proportionally more false-alarm errors in order to improve the detection rate. 90% vs. 10% is actually a fairly benign example of this problem, in many real world cases the ratio is much worse.

Possible solutions

The two common ways to deal with this problem are:

1) Sampling. For each training step, either a) over-sample the underrepresented class, or b) under-sample the overrepresented class.

2) Error-weighting. Penalize misses of the event (model says "no event" when there is one) more than false alarms (model says "event" when there is no event).

Different packages that implement tree-based classification often implement one or both of these approaches out of the box.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.