1
$\begingroup$

Why is it common practice to take the log of the dependent variable Y? To be clear, I understand that under appropriate circumstances that taking the log can help normalize the distribution/linearize the model and I have read other threads discussing this. What I am confused about is why is it okay to transform and normalize/linearize Y and make it falsely 'appear' normal instead of using the real, raw data? Don't we want to train the X variable(s) to be able to predict/determine Y as it is, so why are we altering Y?

Additionally, when exactly in the modeling process do we do this? Would I log Y in the linear model lm(log(Y) ~ X, data = df)? Would I do it when calculating RMSE? Thanks in advance!

$\endgroup$

marked as duplicate by kjetil b halvorsen, whuber regression Jul 17 at 11:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

3
$\begingroup$

Don't we want to train the X variable(s) to be able to predict/determine Y as it is, so why are we altering Y?

Training the model to predict an invertible transform of the outcome as a function of X is the same as training the model to predict Y form X. I don't see this as a problem because we can recover predictions on the Y scale from the predictions on the transformed scale.

Why do we want to make it falsely 'appear' normal instead of using the real, raw data?

Because the normal has nice properties that make computation, interpretation, and inference tractable.

$\endgroup$
0
$\begingroup$

You asked so many questions , I didn't know which one to ignore ... But I trust my discussion here helps you be less befuddled about the role of variance stabilizing transformations.

Optimal Box-Cox transforms should be based upon the residuals from a useful model not necessarily the original series i.e. it should be based upon a conditional ARMAX model structure When (and why) should you take the log of a distribution (of numbers)?. Box - Cox transforms remedies linkage between the expected value of Y and the dispersion in the error process.

Non-constant dispersion/variance in the original data may be caused by unusual values , changes in model parameters over time etc. this these things (first moment) must be settled first.Tests of significance of model parameters are based a set of errors from a model has constant variance thus one needs to empirically validate this.

I should also comment that model error variance change can sometimes be at a particular discret points in time culminating in Weighted Least Squares.. See http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html for a discussion of identifying variance change points and how to rectify this by effectively weighting observations.

If you wanted to post an actual data set (in another question ) then I might be able to walk through an actual analysis.

$\endgroup$
  • $\begingroup$ If you are happy with my answer , please accept it and close the question $\endgroup$ – IrishStat Aug 15 at 8:00
0
$\begingroup$

as @Demetri Pananos pointed out "Training the model to predict an invertible transform of the outcome as a function of X is the same as training the model to predict Y form X.", so havnig a linear model where the dependent variable is the log of your variable of interest does not undermine your ability (once you have fit the model) to predict the dependent variable knowing the independent variable.

however there is no need to do it JUST because the dependent variable is not normally distributed. what needs to be normally distributed in a linear model are the residuals, not each variable (dependent or independent) taken alone.

so, a priori, you should take the log of the dependent variable if for example you think that the independent variable is related linearly to the log of the dependent variable.

$\endgroup$
  • $\begingroup$ If you assume a model form that is non-linear but can be transformed to a linear model such as $\log Y = \beta_0 + \beta_1t$ then one would be justified in taking logarithms of $Y$ to meet the specified model form . This has nothing to do with how X is used or transformed. $\endgroup$ – IrishStat Jul 17 at 19:45
  • $\begingroup$ how is this different from my last sentence? $\endgroup$ – fabiob Jul 23 at 7:08
  • $\begingroup$ no different .... agreed $\endgroup$ – IrishStat Jul 23 at 8:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.