7
$\begingroup$

My target feature is right-skewed. I want to apply gradient boosting regression algorithm to predict it but I'm not sure what kind of preprocessing should I apply.

As gradient boosting is based on decision trees the common intuition declares that logarithmic transformation won't help much. Another way is to remove outliers based on a threshold and to look at the performance of the model. But I'm not sure this is an optimal solution.

Any suggestion?

$\endgroup$
3
  • 3
    $\begingroup$ Removing outliers is a good way to bias your trained model. Do not remove any data. What makes you think a transform of the data will not help? $\endgroup$ Nov 14, 2018 at 14:24
  • 2
    $\begingroup$ @DemetriPananos Generally transformation works fine. But as the majority of gradient boosting algorithm used CART trees under the hood which are invariant to scale changes so a log transform should not change the resulting tree. Correct me if I wrong $\endgroup$ Nov 14, 2018 at 14:28
  • $\begingroup$ @DanielChepenko algo is invariant to scale changes but the outcome is still worse when you don't apply preprocessing eg log transform when you can/should. Would you rather have the model learn a straight line or some complex curvy relationship? If you can help the model learn to make better predictions with less parameters using human/domain expertise, that sounds like a smart thing to do. $\endgroup$
    – citynorman
    Mar 17, 2020 at 23:49

1 Answer 1

5
$\begingroup$

I think we must first consider if the outliers are "true data" or just simply noise/corrupted input.

If they are corrupted data (e.g. an adult human weighting 775 kg) then it is perfectly reasonable to exclude these instances from further analysis. If these instances though are reasonable data we might want to work with them, rather than around them. A first obvious fix that does not involve data transformations, would be to employ a custom objective function approximating a MAE, a Huberised loss or a quantile loss. That would allow minimising the effect of instances that might seem highly unnatural. In general and without containing yourself to gradient boosting, I would suggest looking into robust statistics to get a better idea of how one would classically deal with potentially noisy and/or skewed data (for example using a GAM with a scaled-T distribution for the family of the response).

As you say, potentially transforming then back-transforming your data (log(x+1) being a common choice of strictly non-negative data) is also a potentially reasonable approach. Go for it, just do not get too crazy because while model interpretability is not a prime concern when predicting, if the transformation is just too convoluted (e.g. through some arbitrary power tranformation), debugging and/or improving an existing model becomes even more complicated than it should.

Finally, I would suggest you look into some data competitions that are concerned with skewed variables themselves (e.g. the Allstate Insurance claims severity predictions), these guys have some nifty ideas too!

$\endgroup$
5
  • $\begingroup$ Hi! Sorry for my late response, but your question was really helpful. I even tried to solve the competition you posted and got some good results. $\endgroup$ Dec 8, 2018 at 13:48
  • $\begingroup$ Taking as an example this Allstate competition - how should I denormolize data after training. The target value is skewed hence I applied log-transformation. After that I fitted the model, but it seems that I need to take exponent before submission, as the test data has different scale. Is it a correct approach $\endgroup$ Dec 8, 2018 at 13:51
  • $\begingroup$ Yes, you are correct; it is perfectly reasonable to exponentiate the final estimate before submission if during training we used the logarithms of the response variable. That is because, while we projected the data into a space where we could solve the problem more efficient (the log-domain in this particular case), our estimate will be used in their original domain (thus we need to exponentiate them). $\endgroup$
    – usεr11852
    Dec 8, 2018 at 14:01
  • 1
    $\begingroup$ Our intention is to estimate E[Y], but after log transformation and denomalization we would get exp(E[log(Y)]). These two quantities are not the same. In particular the former is larger than the latter due to Jensen's inequality. So after denormalization, our prediction is not what we thought it would be. Will this affect the model performance? $\endgroup$
    – Pan
    May 14, 2020 at 20:43
  • $\begingroup$ Yes, it will; indeed $\log(E\{x\})$ is greater or equal to $E\{\log(x)\}$. This issue especially prevalent when forecasting. If we assume a log-normal for example we should estimate our mean as $\exp(\hat{\mu} + \frac{\hat{\sigma}^2}{2})$. We have to look into doing a Taylor expansion around $x_0 = \hat{\mu}$. $\endgroup$
    – usεr11852
    May 14, 2020 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.