How to deal with predictions if taking log of dependent variable

Question

I have a very basic question about linear regression. I have a dataset where the response variable is largely skewed to the right -- if I take a log of it, the distribution becomes a lot closer to normal which should help in terms of prediction (otherwise, the observations out in the tail are always off by a considerable amount).

However, if I take the log of the y variable in my training data set, how do I handle the same variable for the test data set? Do I take the log of this too? Is there something that needs to be done later to get the observations back to their true state?

I'm a beginner in dealing with these types of transformations so any advice would be appreciated.

Answer 1

Yes, you need to apply the transformation to both test and training set when you transform; you are no longer fitting a regression to "y", you are fitting a regression to "log(y)".

For predictions from your model, if you would like to express them on the same scale as the original variable, you need to "back transform" which means applying the inverse of the transformation. For a natural log transform, that means e^(log(y)).

If you are calculating, say, confidence intervals using standard errors, it is important to do this transformation last. That is, you cannot get confidence intervals by multiplying ~1.96 by the back-transformed standard error. You always want to complete your work on the transformed scale data, and only convert back to the original scale for reporting final results.

For some back transformations, you may need more caution, see Back transforming regression results when modeling log(y)