Why is it okay to take the log (or any other transformation) of the dependent variable? [duplicate]

Question

Why is it common practice to take the log of the dependent variable Y? To be clear, I understand that under appropriate circumstances that taking the log can help normalize the distribution/linearize the model and I have read other threads discussing this. What I am confused about is why is it okay to transform and normalize/linearize Y and make it falsely 'appear' normal instead of using the real, raw data? Don't we want to train the X variable(s) to be able to predict/determine Y as it is, so why are we altering Y?

Additionally, when exactly in the modeling process do we do this? Would I log Y in the linear model lm(log(Y) ~ X, data = df)? Would I do it when calculating RMSE? Thanks in advance!

Answer 1

Don't we want to train the X variable(s) to be able to predict/determine Y as it is, so why are we altering Y?

Training the model to predict an invertible transform of the outcome as a function of X is the same as training the model to predict Y form X. I don't see this as a problem because we can recover predictions on the Y scale from the predictions on the transformed scale.

Why do we want to make it falsely 'appear' normal instead of using the real, raw data?

Because the normal has nice properties that make computation, interpretation, and inference tractable.

Answer 2

You asked so many questions , I didn't know which one to ignore ... But I trust my discussion here helps you be less befuddled about the role of variance stabilizing transformations.

Optimal Box-Cox transforms should be based upon the residuals from a useful model not necessarily the original series i.e. it should be based upon a conditional ARMAX model structure When (and why) should you take the log of a distribution (of numbers)?. Box - Cox transforms remedies linkage between the expected value of Y and the dispersion in the error process.

Non-constant dispersion/variance in the original data may be caused by unusual values , changes in model parameters over time etc. this these things (first moment) must be settled first.Tests of significance of model parameters are based a set of errors from a model has constant variance thus one needs to empirically validate this.

I should also comment that model error variance change can sometimes be at a particular discret points in time culminating in Weighted Least Squares.. See http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html for a discussion of identifying variance change points and how to rectify this by effectively weighting observations.

If you wanted to post an actual data set (in another question ) then I might be able to walk through an actual analysis.

Answer 3

as @Demetri Pananos pointed out "Training the model to predict an invertible transform of the outcome as a function of X is the same as training the model to predict Y form X.", so havnig a linear model where the dependent variable is the log of your variable of interest does not undermine your ability (once you have fit the model) to predict the dependent variable knowing the independent variable.

however there is no need to do it JUST because the dependent variable is not normally distributed. what needs to be normally distributed in a linear model are the residuals, not each variable (dependent or independent) taken alone.

so, a priori, you should take the log of the dependent variable if for example you think that the independent variable is related linearly to the log of the dependent variable.