I was reading a forum post on Kaggle here. The person is showing an example of exploratory analysis for a competition.

During the exploratory analysis one of the charts generated is a log(targetVariable) distribution. Here is what it looks like: enter image description here

I generated the same chart based on not log of the data just for my interest:

ggplot(full) + geom_histogram(mapping=aes(x=loss))

And it looks like this: enter image description here

So what? Now what? Why do this? Two facts have been learned:

  1. The target variable is skewed left (Is that correct?) (It is not correct, see comment below)
  2. The log of the target variable is normally distributed

In the context of multiple linear regression, how does this inform my decisions on prediction? Do I have to log() all my predictor variables now? Why? Other than generating a nice shiny chart, how does exploratory analysis of the distribution of thew target variable help when it comes to predicting the target variable?

  • $\begingroup$ FYI the convention for "sense of skew" is towards the tail, so your distribution is right skewed. If I understand your post correctly, then you have not presented any information on your predictors, so your exploratory analysis says nothing about whether they should be log-transformed or not. (This is a separate question from whether the target variable is lognormal.) $\endgroup$ – GeoMatt22 Oct 18 '16 at 2:28
  • $\begingroup$ @GeoMatt22 thanks for the info. I will not edit the question so any other learners get this tidbit too $\endgroup$ – Doug Fir Oct 18 '16 at 2:45

In this case, the y variable has a noticeable positive skew. It's hard to tell from the relatively low-resolution figure of the original variable, but it's possible that there is an abrupt cutoff just to the left of the mode.

This might be an indication of a variable that has an artificial left truncation. The "true" value of an employee to a workplace, for example, might range from high positive values to even negative values. If you try to ascertain this from paid wages, though, you will encounter an artificial left cutoff in the form of a minimum wage. Taking a logarithm of this type of variable can make it more normally distributed (as in the case you show). Say you're considering the effect of number of school years and recorded felonies on wages. The true model might be one that includes values that have been artificially been truncated.

This, however, leads to the question of whether taking the logarithm will improve the regression's performance. Saying that an increase in $x$ will cause a linear increase in $y$, is very different than saying that an increase in $x$ will cause a linear increase in $log(y)$ - these are contradictions, in fact.

Linear Regression Models with Logarithmic Transformations describes the four conceptual possibilities of taking logarithms in regression (a logarithm can be taken on a dependent and/or independent variable).

I think the important point is that the decision of whether to take a logarithm of the dependent variable depends not only on it, but also on the dependent variable(s). For one, as the paper above explains, it really depends on what the true model is. Also, note that, to some extent, linear regression assumes that the residual is normally distributed, and the residual depends on transformations to all variables.

So while the graphs you show are some indication that a log transformation on the y might be beneficial (it makes it more normal), you can't fully decide that based on the y alone.

| cite | improve this answer | |
  • $\begingroup$ Thanks for the details, and for the linked pdf which was useful. $\endgroup$ – Doug Fir Oct 18 '16 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.