How to handle Regression data thats not linear

I'm new to stats and am using Python 2.7 to fit a regression model (Random Forest). When I plot the percentile plot of the prices before and after a log transformation, I get

Percentile plot (Left: No transformation. Right: Log transformed)

I notice that the plot became more linear, except at the beginning where it's a little strange. What will you do under such circumstances, and what may have caused that abnormality?

When I plot only the prices that have higher values (> 400000), the plot becomes

Is it sufficiently linear now?

What does a linear plot imply?

How should I treat values under 400000 which I have excluded from being used to fit the regression model?

How do you decide whether to log transform the variables too? By doing a similar percentile plot?

Quantile-normal plot (Log transformation of y variable)

Left: Training Set. Right: Test Set

Quantile-normal plot ( No transformation of y variable)

Left: Training Set. Right: Test Set

Why does log transformation not do anything?

• I don't know why you squared the residual, but the log transform always does something, so something is wrong in your code somewhere. – Peter Flom May 14 '13 at 10:15
• @PeterFlom Ah I discovered I have seriously confused myself! Updated the question with the correct plots without squaring the residuals. Seems like the r^2 became lower for both the training and testing set after doing the log transform. I'm guessing the log transform does not really help, in this case is there another transformation that I can use? Can you deduce anything from the shape of the quantile normal plot? – Nyxynyx May 15 '13 at 2:49
• The fact that $R^2$ became lower isn't relevant; you are testing the assumptions. It is easy to come up with a data set where $R^2$ is very high but model is not right. – Peter Flom May 15 '13 at 10:12