1
$\begingroup$

I have a variable that I want to model, which has a skewed distribution. Log transforming the var gives is a normal-like distribution. When training a Random Forest regressor on the non-transformed var, I get worse performance than when I log-tranform the var. I am bit puzzled about whether I should do this knowning that the random forest regressor is predicting the mean of the leafs. If trained on a log tranformed var, that means that the prediction is the mean of the logs of the values in the leafs. Which (when tranformed back) is not equal to the mean of the real values.

Any opinion?

$\endgroup$
1
$\begingroup$

I will be assuming that by "better performance" you mean better CV/validation performance, and not train one.

I want to invite you to think of what the effect of log-transforming the target variable is on single regression trees

Regression trees make splits in a way that minimizes the MSE, which (considering that we predict the mean) means that they minimize the sum of the variances of the target in the children nodes.

What happens if your target is skewed?
If your variable is skewed, high values will affect the variances and push your split points towards higher values - forcing your decision tree to make less balanced splits and trying to "isolate" the tail from the rest of the points.

Example of a single split on non-transformed and transformed data:

As a result overall, your trees (and so on RF) will be more affected by your high-end values if your data is not transformed - which means that they should be more accurate in predicting high values and a bit less on the lower ones.

If you log-transform you reduce the relative importance of these high values, and accept having more error on those while being more accurate on the bulk of your data. This might generalize better, and - in general - also makes sense. Indeed in the same regression, predicting $\hat{y}=105$ when $y=100$ is better than predicting $\hat{y}=15$ when $y=11$, because the error in relative terms often matters more than the absolute one.

Hope this was useful!

$\endgroup$
  • $\begingroup$ This is super useful. idd by performance I am referring to CV. My only concern has to do with with the averaging of the log-transformed values but I do see the overall point of transforming to mitigate your estimator putting more emphasis on the overall performance rather than the densest part of the distribution $\endgroup$ – LetsPlayYahtzee Feb 6 at 12:28
  • $\begingroup$ it's the opposite - by logtrasforming you try to have a more general performance and focus "less" on the tails and more on the dense part of the distribution. And it's not just for regression trees, the same works for any regression problem as far as I know. $\endgroup$ – Davide ND Feb 6 at 13:13
  • $\begingroup$ @LetsPlayYahtzee - that follows the same problem - the average of the log is not the log of the averages, and happily so: averaging the log values rather than taking the log of the averages is LESS sensitive to tail values, and therefore is in line with the choice that you make of log transforming your target in order to make the skewness less impactful of the results $\endgroup$ – Davide ND Feb 6 at 20:49
3
$\begingroup$

Tangentially, the marginal distribution (that is, the distribution obtained when plotting a histogram) of the outcome is irrelevant in regression since most regression methods make assumptions about the conditional distribution (that is, the distribution obtained when plotting the histogram of the outcome were I to only observe outcomes which have the same features). Now, on to your question.

If you are evaluating the performance of on the transformed outcome, the results can be misleading. Because the log essentially squeezes the outcomes, the variance is also shrunk meaning predictions will be closer to the observations. This shrinks the loss and appears to make your model better. Try doing this

from sklearn.dummy import DummyRegressor
from sklearn.model_selection import cross_val_score

cross_val_score(DummyRegressor(), X, y, scoring = 'neg_mean_squared_error')
cross_val_score(DummyRegressor(), X, np.log(y), scoring = 'neg_mean_squared_error')

Same data, but the scores are immensely different. Why? Because the log shrinks the variance of the outcomes making the model appear better even though it does nothing different.

If you want to transform your outcome, you can:

  • Train the model on the transformed outcomes
  • Predict on a held out set
  • Re-transform the predictions to the original space
  • Evaluate the prediction quality in the original space

Sklearn makes this very easy with their TransformedTargetRegressor.

from sklearn.ensemble import RandomForestRegressor
from sklearn.compose import TransformedTargetRegressor
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_regression

import numpy as np

rf = RandomForestRegressor()
log_rf = TransformedTargetRegressor(rf, func = np.log, inverse_func=np.exp)


params = {'regressor__n_estimators': [10,100,1000]}


gscv = GridSearchCV(log_rf, param_grid=params,refit = True)

X,y = make_regression(n_samples = 10_000, n_features=50, n_informative=5)
y -= y.min()-1 #Make the outcome positive.

Xtrain, Xtest, ytrain, ytest = train_test_split(X,y, test_size = 0.25)

gscv.fit(Xtrain, ytrain)

This will ensure that the model is trained on the log-transformed outcomes, back transforms into the original space, and evaluates the loss in the original space.

$\endgroup$
  • $\begingroup$ that's exactly what I am doing! My question has to do with the fact that the RF regressor will average log-transformed values, that I then convert back to the original space. But the mean of the transformation when (converted back to the original space) is not equal to the mean in the original space. $\endgroup$ – LetsPlayYahtzee Feb 4 at 18:31
  • $\begingroup$ I would encourage you to predict, transform, then compute the loss rather than predict, compute the loss, then transform. Does that make sense? $\endgroup$ – Demetri Pananos Feb 4 at 18:33
  • $\begingroup$ That's what I am doing :) The performance (on the original var) after training on the log-transformed var and projecting back, is better. But I can't tell if this is by accident - since as I mentioned above the RF regressor is applying a mean on the leaf values. $\endgroup$ – LetsPlayYahtzee Feb 4 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.