Log-Transforming target var for training a Random Forest Regressor

Question

I have a variable that I want to model, which has a skewed distribution. Log transforming the var gives is a normal-like distribution. When training a Random Forest regressor on the non-transformed var, I get worse performance than when I log-tranform the var. I am bit puzzled about whether I should do this knowning that the random forest regressor is predicting the mean of the leafs. If trained on a log tranformed var, that means that the prediction is the mean of the logs of the values in the leafs. Which (when tranformed back) is not equal to the mean of the real values.

Any opinion?

Answer 1

I will be assuming that by "better performance" you mean better CV/validation performance, and not train one.

I want to invite you to think of what the effect of log-transforming the target variable is on single regression trees

Regression trees make splits in a way that minimizes the MSE, which (considering that we predict the mean) means that they minimize the sum of the variances of the target in the children nodes.

What happens if your target is skewed?
If your variable is skewed, high values will affect the variances and push your split points towards higher values - forcing your decision tree to make less balanced splits and trying to "isolate" the tail from the rest of the points.

Example of a single split on non-transformed and transformed data:

As a result overall, your trees (and so on RF) will be more affected by your high-end values if your data is not transformed - which means that they should be more accurate in predicting high values and a bit less on the lower ones.

If you log-transform you reduce the relative importance of these high values, and accept having more error on those while being more accurate on the bulk of your data. This might generalize better, and - in general - also makes sense. Indeed in the same regression, predicting $\hat{y}=105$ when $y=100$ is better than predicting $\hat{y}=15$ when $y=11$, because the error in relative terms often matters more than the absolute one.

Hope this was useful!

Answer 2

Tangentially, the marginal distribution (that is, the distribution obtained when plotting a histogram) of the outcome is irrelevant in regression since most regression methods make assumptions about the conditional distribution (that is, the distribution obtained when plotting the histogram of the outcome were I to only observe outcomes which have the same features). Now, on to your question.

If you are evaluating the performance of on the transformed outcome, the results can be misleading. Because the log essentially squeezes the outcomes, the variance is also shrunk meaning predictions will be closer to the observations. This shrinks the loss and appears to make your model better. Try doing this

from sklearn.dummy import DummyRegressor
from sklearn.model_selection import cross_val_score

cross_val_score(DummyRegressor(), X, y, scoring = 'neg_mean_squared_error')
cross_val_score(DummyRegressor(), X, np.log(y), scoring = 'neg_mean_squared_error')

Same data, but the scores are immensely different. Why? Because the log shrinks the variance of the outcomes making the model appear better even though it does nothing different.

If you want to transform your outcome, you can:

• Train the model on the transformed outcomes
• Predict on a held out set
• Re-transform the predictions to the original space
• Evaluate the prediction quality in the original space

Sklearn makes this very easy with their TransformedTargetRegressor.

from sklearn.ensemble import RandomForestRegressor
from sklearn.compose import TransformedTargetRegressor
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_regression

import numpy as np

rf = RandomForestRegressor()
log_rf = TransformedTargetRegressor(rf, func = np.log, inverse_func=np.exp)


params = {'regressor__n_estimators': [10,100,1000]}


gscv = GridSearchCV(log_rf, param_grid=params,refit = True)

X,y = make_regression(n_samples = 10_000, n_features=50, n_informative=5)
y -= y.min()-1 #Make the outcome positive.

Xtrain, Xtest, ytrain, ytest = train_test_split(X,y, test_size = 0.25)

gscv.fit(Xtrain, ytrain)

This will ensure that the model is trained on the log-transformed outcomes, back transforms into the original space, and evaluates the loss in the original space.