I have a question regarding the normality of predictors. I have 100,000 observations in my data. The problem I am analysing is a classification problem so 5% of the data is assigned to class 1, 95,000 observations assigned to class 0, so the data is highly imbalanced. However the observations of the class 1 data is expected to have extreme values.

  • What I have done is, trim the top 1% and bottom 1% of the data removing, any possible mistakes in the entry of such data)
  • Winsorised the data at the 5% and 95% level (which I have checked and is an accepted practise when dealing with such data that I have).

So; I plot a density plot of one variable after no outlier manipulation enter image description here

Here is the same variable after trimming the data at the 1% level enter image description here

Here is the variable after being trimmed and after being winsorised enter image description here

My question is how should I approach this problem.

First question, should I just leave the data alone at trimming it? or should I continue to winsorise to further condense the extreme values into more meaningful values (since even after trimming the data I am still left with what I feel are extreme values). If I just leave the data after trimming it, I am left with long tails in the distribution like the following (however the observations that I am trying to classify mostly fall at the tail end of these plots). enter image description here

Second question, since decisions trees and gradient boosted trees decide on splits, does the distribution matter? What I mean by that is if the tree splits on a variable at (using the plots above) <= -10. Then according to plot 2 (after trimming the data) and plot 3 (after winsorisation) all firms <= -10 will be classified as class 1.

Consider the decision tree I created below.

enter image description here

My argument is, regardless of the spikes in the data (made from winsorisation) the decision tree will make the classification at all observations <= 0. So the distribution of that variable should not matter in making the split? It will only affect at what value that split will occur at? and I do not loose too much predictive power in these tails?

  • 6
    $\begingroup$ None of these algorithms makes any assumptions about the distribution of the predictors. Further, trees make piecewise constant predictions, so are not particularly susceptible to high leverage points in the way something like linear regression is. So overall, I think you need to describe in more detail what features of the problem caused you to take these steps. $\endgroup$ May 9 '18 at 20:24
  • 7
    $\begingroup$ Also, I'm a broken record on this point, but 5% vs. 95% is not highly imbalanced, it's more or less a normal, everyday, balance. You shouldn't really have to take any steps to "deal" with this. Just work on the level of probabilities, avoid algorithms that try to make hard class assignments; if you need a decision rule, pick your threshold intelligently. $\endgroup$ May 9 '18 at 20:25
  • 6
    $\begingroup$ Hmmm. I am sad to hear that most of the economics literature perpetuates that myth. As long as you use gradient boosting to predict probabilities, you should not have to balance your classes. I believe Frank Harrel talks about this a lot in the context of medicine, so check out his writing if your are interested. $\endgroup$ May 9 '18 at 20:50
  • 6
    $\begingroup$ The standard advice around here is that you dont have an imbalanced problem until you have proven that you do. I would not use the class weight argument. Xgboost should give you predicted probabilities calibrated to your data, and you should be evaluating the success of your model based on these, not something based on the confusion matrix. Confusion matrices are for evaluating decision rules, not models. $\endgroup$ May 9 '18 at 23:34
  • 4
    $\begingroup$ You actually give a compelling reason not to truncate any variables in your opening paragraph. You have a prior belief that class 1 observations will take extreme values. If your goal is to distinguish the two classes, why would you ever want to hide this information from the model? Xgboost could learn that extreme values occur in class 1, and improve its predictions. $\endgroup$ May 17 '18 at 14:52

Second question. Yes, algorithms based on decision trees are completely insensitive to the specific values of predictors, they react only to their order. It means that you don't have to worry about "non-normality" of your predictors. Moreover, you can apply any monotonic transformation to your data, if you want - it will not change predictions of decision trees at all!

First question. I feel you should leave your data alone. By trimming and winsorizing it, you discard information that might be meaningful for your classification problem.

For linear models, long tails introduce noise that may be harmful. But for decision trees it is not a problem at all.

If you are too afraid of long tails, I would suggest to apply a transformation to your data, that puts it into a prettier scale without distorting the order of your observations. For example, you can make the scale logarithmic by applying $$ f(x) = \text{sign}(x) \alpha\log(|x / \alpha|+1) $$

For small x (roughly from $-\alpha$ to $\alpha$), this function is close to identity, but large values are heavily shrunk towards 0, but monotonicity is strictly preserved - thus, no information is lost.

How can removing extremes affect quality of prediction? By removing extreme values, you indeed can prevent your model for making splits in very high or very low points. This restriction leads exactly to non-increasing of the ability of your model to fit the training data. You have quite a large dataset (100K of points is quite much, if it is not very high-dimensional), so I assume that your model doesn't suffer from severe overfitting, if you regularize it properly (e.g. by controlling maximum tree size and number of trees). If it is the case, then restricting the model from splitting in high or low points will lead to degeneration of prediction quality on the test set as well.

  • $\begingroup$ Thanks for your comment! Regarding the second question (which is related to what I commented about above just now). Will keeping the extreme values in the data affect the specific values the model makes the decision to split on? i.e. If I kept extreme values and the model decides to split on a value of <=10.5 for one variable in one model and in the second model (removing extreme values), the model may then decide to split on <=10.0, thus having a difference of 0.5 which could be the difference in obtaining higher Specificity scores and correctly predicting class 1 observations. $\endgroup$
    – user113156
    May 18 '18 at 23:08
  • $\begingroup$ I am looking at this from a more theoretical perspective, just to obtain a better understanding. $\endgroup$
    – user113156
    May 18 '18 at 23:09

It will sound like circular reasoning, but: the method that scores best on the evaluation criterion is the best method.

Instead of worrying about the most theoretically sound approach, answer: given two approaches, how would you decide which one is best? When you’ve reduced the decision process to a quantitative algorithm, then every idea you have is potentially worth trying out, and the idea that achieves the best outcome is the best idea.

Then the focus is on ensuring the evaluation process leads to valid outcomes (apply cross-validation, calculate confidence intervals, possibly correct for false discovery rate).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.