How to get the actual mean absolute error in cross validation after transforming the target variable y?

Question

For a target variable y, it is transformed using np.log1p. Then a random forest regression model is trained using the transformed y.

Then I tried to use cross_val_score in sklearn to compute the neg_mean_absolute_error. The mean of the returned scores is -0.17. Should I convert it back by using np.emp1(0.17)?

But the error is too small. As I am predicting housing price. The error should be much larger than this.

Answer 1

Rather than transforming the estimate of the generalization error, you're better off using a model which transforms the training data and then is capable of predicting back on the original scale.

This is done with sklearn.compose.TransformTargetRegressor as shown below.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.compose import TransformedTargetRegressor
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

N = 1000
x = np.random.normal(size = (N, 1))
y = 10 + x + np.random.normal(size = (N, 1))
y = y.ravel()

model = Pipeline([
    ('ttr', TransformedTargetRegressor(regressor=LinearRegression(), func = np.log1p, inverse_func = lambda x: np.exp(x)-1))
])


cross_val_score(model, x, y, cv=5, scoring = 'neg_mean_squared_error').mean()

>>>-0.997... #Depending on the data used.

Note that our cross validated error is very close to the true standard deviation of the noise (which is 1 in this case) despite the model mispecification (the true conditional mean is linear whereas the model is exponential)

Answer 2

You should be able to calculate the mean absolute error (MAE) using just very basic functions in Python. If y is your target variable and you used your transformed target variable y_trans to train the model, you will get a transformed outcome variable out_trans. To get a meaningful MAE, in housing price e.g., you would have to transform your outcome variable,out_trans, using the inverse transformation function you used in y. Note that this is different from transforming the MAE itself, as you suggested. After that, the MAE is just the mean of the differences between y and out, in module:

import numpy as np
mae = np.mean(abs(y) - abs(out))

You could of course compute the MAE with the transformed quantities, but as you said, the unit won't be that meaningful.