I'm new to stats and am using Python 2.7 to fit a regression model (Random Forest). When I plot the percentile plot of the prices before and after a log transformation, I get

Percentile plot (Left: No transformation. Right: Log transformed)

enter image description here

I notice that the plot became more linear, except at the beginning where it's a little strange. What will you do under such circumstances, and what may have caused that abnormality?

When I plot only the prices that have higher values (> 400000), the plot becomes

enter image description here

Is it sufficiently linear now?

What does a linear plot imply?

How should I treat values under 400000 which I have excluded from being used to fit the regression model?

How do you decide whether to log transform the variables too? By doing a similar percentile plot?

Quantile-normal plot (Log transformation of y variable)

Left: Training Set. Right: Test Set

enter image description here

Quantile-normal plot ( No transformation of y variable)

Left: Training Set. Right: Test Set

enter image description here

Why does log transformation not do anything?

  • $\begingroup$ I don't know why you squared the residual, but the log transform always does something, so something is wrong in your code somewhere. $\endgroup$ – Peter Flom May 14 '13 at 10:15
  • $\begingroup$ @PeterFlom Ah I discovered I have seriously confused myself! Updated the question with the correct plots without squaring the residuals. Seems like the r^2 became lower for both the training and testing set after doing the log transform. I'm guessing the log transform does not really help, in this case is there another transformation that I can use? Can you deduce anything from the shape of the quantile normal plot? $\endgroup$ – Nyxynyx May 15 '13 at 2:49
  • $\begingroup$ The fact that $R^2$ became lower isn't relevant; you are testing the assumptions. It is easy to come up with a data set where $R^2$ is very high but model is not right. $\endgroup$ – Peter Flom May 15 '13 at 10:12

You seem to be confusing some things.

1) Regression does not require normally distributed data, it assumes normally distributed errors (which you approximate by residuals)

2) The plots you give don't give good evidence of normality or non-normality; try a quantile-normal plot

3) A single variable can't be linear or not linear; linearity is a quality of relationships among variables.

| cite | improve this answer | |
  • $\begingroup$ Thanks Peter, I've created the quantile-normal plot for both transformed and un-transformed data. With the r^2 that is much higher with log transformation, is this a clear hint that I should do the transformation? Or is the higher r^2 due to the outliers $\endgroup$ – Nyxynyx May 11 '13 at 18:26
  • $\begingroup$ Is that on the residuals? $\endgroup$ – Peter Flom May 11 '13 at 21:45
  • $\begingroup$ Updated question with the quantile-normal plot of the residuals squared. There is no difference between log and no log transformation of the y variable. Is this something that you'll expect? $\endgroup$ – Nyxynyx May 14 '13 at 5:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.