1
$\begingroup$

I am running an XGBoost model with a continuous target variable. With ~200 features I am getting a Test $R^2$ of 0.54. By looking at the distribution of the target variable, it appears it's highly left skewed. So, I took log transformation on the target variable and reran the model which gave me an $R^2$ of 0.68, which is significantly better than the non-transformed model. Then, I reverse transformed the predicted values (antilog) and calculated $R^2$ using original values of the target variable. The $R^2$ is -0.02.

I am having a difficult time wrapping my head around such results. I understand taking antilog (or exponent) will significantly shoot up the error

$log(y) = f(x) + error$

$y = exp(f(x) + error)$

I am trying to understand which results should I trust and if $R^2$ is the correct accuracy metric to look at?

Case 1: Use the model with original non-transformed data ($R^2 = 0.54$)

Case 2: Use the model with log-transformed target variable ($R^2 = 0.68$, but when reverse transformed $R^2 = -0.02$)

$\endgroup$
8
  • 1
    $\begingroup$ The negative $R^2$ should come as no surprise: If you model a transformed variable, there is no reason why the estimated relationship would hold for the original variable. The variance explained in the original variable is meaningless. $\endgroup$ Dec 10, 2019 at 3:01
  • $\begingroup$ How are you fitting the second model (the one with multiplicative error)? $\endgroup$
    – Glen_b
    Dec 10, 2019 at 5:02
  • $\begingroup$ The "distribution of the target variable [appearing] highly left skewed" was your rationale for log-transforming the target variable, but you should be looking at the distribution of your residuals to determine whether a log-transformation is appropriate or not. Are the residuals from your fitted model $y = f(x) + \epsilon$ similarly highly left-skewed? $\endgroup$
    – psboonstra
    Dec 10, 2019 at 15:27
  • $\begingroup$ @FransRodenburg At the end of the day I am interested in predicting the original variable. Log transformation was just to help me create a better model. What would be a fitting accuracy measure to test on the original variable and how much I can trust on 0.68 R2? Which model would you suggest to go ahead with? $\endgroup$ Dec 10, 2019 at 21:59
  • $\begingroup$ @Glen_b-ReinstateMonica I am using XGBoost for prediction of the second model where I use log of target variable. $\endgroup$ Dec 10, 2019 at 22:00

2 Answers 2

3
$\begingroup$

I used to deal with a similar problem few years ago. When you inverse the log-treansform, 0.2 unit difference in log scale may be 1000 in real (inversed) value.

The best practice to maximize your metric is to develop a model using the transformed variable BUT calculate the metric on real (inversed) variable.

As a result, you may need to use a customized scoring function if you use GridSearchCV() of sklearn package.

To make a scorer, you can use something like following code:

def r2new(y_true, y_pred):
    return r2_score(np.exp(y_true), np.exp(y_pred))

r2new_scorer = make_scorer(r2new, greater_is_better=True)

xgb_clf.GridSearchCV(..., scoring=r2new_scorer)

y_log=np.log(y)
xgb_clf.fit(X,y_log)
y_log_calc=xgb_clf.predict(X)
print("R2: %s"%r2_scorer(y, np.exp(y_log_calc)))
$\endgroup$
1
$\begingroup$

In general Case 2 is the best option.

Considering that $R^2$ is dependent from the sum of squared residuals, it is highly affected by high values in skewed distributions.
Assuming your dataset has five points with $y = [10, 12, 15, 13, 105]$, then two models with predictions $\hat{y}=[5,7,10,8,105]$ and $\hat{y}=[10,12,15,13,115]$ would have the same $R^2$ with a sum of squared residuals of $100$, even though the errors of the first are much bigger in relative terms! (of course this is an exagerated example with an outlier, but works the same for skewed distributions).

When you transform to the log, values become more comparable and you reduce the relative importance of tail observations!

Finally, it all boils down to what your data represents and what type of error you want to minimize. If missing a prediction by 10 points is the same to you both for a true value of 100 and 10000, then use non-transformed variables. If (like in most cases) you care more about making close prediction on the bulk of the data rather than the tail, then go with the log transformation.

$\endgroup$
1
  • $\begingroup$ Thanks for the clear explanation. This is very helpful. Perhaps it is a better idea to build a separate model for tail observations. $\endgroup$ Dec 12, 2019 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.