# Does the non-normality matter in using regression for prediction?

Does the non-normality matter in using regression for prediction?

Hi all,

In the QQ plot of the residuals after linear regression, the residuals turned out to be highly non-Gaussian.

Most of the points (95%) are below the 45 degree straightline. And those below the straightline are all on the lower side.

The shape of the curve looks like f(x)=x^(1/5) for x on [0, 1]. (The 5% points in the middle of this curve are above the 45 degree straight line).

It turns out that the dependent variable y has data that are highly non-Gaussian. They are all in [0, 1], but mostly clustered around 1.

So I tried various ways of transforming y. The latest one I've found was to do y_new = y ^ 7 .

Using this transformation, the y_new data become much more symmetrical than before but they are still no way look like Gaussian in histogram.

Being very disappointed, I don't have any more weapon in my bag...

Instead, I began to wonder, if my end goal is just to get the yhat, i.e. the prediction of the data on a wider data-set,

does the Non-normality even matter, in terms of accuracy for prediction?

Please shed some lights on me.

Thank you!

The y data looks very similar to the data generated using the following script:

mydata=rt(10000, df=5)
mydata=mydata[mydata<0.8]
mydata=(mydata-min(mydata))/(max(mydata)-min(mydata))

hist(mydata, 100)

-
If you could provide more information about y maybe we could be of more assistance. In short tailed cases the nonnormality is not as severe a problem as in heavytailed case. Have you tested your regression model on data that wasn't used in the fitting process? –  Michael Chernick Jul 6 '12 at 20:59
Yes, I've tried that. I have also updated my question, hopefully it will answer your question better. Thanks a lot! –  Luna Jul 6 '12 at 21:27
I disagree with AdamO and Greg Snow on the general statement that nonnormality does not hurt prediction. For heavytailed distributions at least, least squares is overly sensitive to outliers and leverage points. This affects the accuracy of the regression coefficients and consequently your prediction accuracy. Assume the form of the model is correct and the residuals have zero mean and constant variance but are not normal then the Gauss-Markov theorem says that they will still be minimum variance among unbiased estimators. In that sense the estimates are still good. –  Michael Chernick Jul 6 '12 at 21:34
I think that is the motivation that AdamO and Greg Snow to say that they are still good for predicting new values of y. But the best predictor need not always be unbiased. Robust regression can produce better estimates when the distribution of the residuals are heavytailed. But this does not seem to be you problem. –  Michael Chernick Jul 6 '12 at 21:39
Thanks Mike. Is my data heavy-tailed or not? Thank you! –  Luna Jul 6 '12 at 23:06

Normality is not needed to fit the regression line, the normality is used when making inference about it (hypothesis tests and confidence intervals). Without normality the regression equation will still predict the mean (assuming the correct model), but a prediction interval based on normality and that mean will also be meaningless. However, in highly skewed cases the mean may not be the most meaningful statistic, which is why things like robust regression, quantile regression, and others are often suggested. With a y value between 0 and 1 you might consider Beta regression.

Most of this comes down to figuring out what question you are trying to answer and what makes sense relative to the question and the science (does $y^7$ have any scientific meaning? if not I would avoid using it).

-
Thanks a lot! I have updated my question with some code to show what the y data look like. In theory, the y data may not be always in [0, 1]. However, in reality, I found that min(y) = -0.06393158, max(y)=0.9942669... can I still use Beta regression? What's the benefit of using Beta regression vs. Linear regression? More accurate? Thank you! –  Luna Jul 6 '12 at 21:32
Beta regression assumes that the response comes from a Beta distribution (like gaussian regression assumes a normal). So it is only for data between 0 and 1 but works well for things like proportions when you don't have the denominator to do logistic regression. –  Greg Snow Jul 6 '12 at 21:50

As I told you with your previous question the prediction can be highly influenced by nonnormality and robust regression is an alternative. It sounds like the nonnormality is due to very short tails and heavy skewness. You say that your DV is confined to [0,1] with high concentration at 1. That certainly is short tailed and it would be skewed if there are a few point far away from 1. If y were just a binary response only taking the values 0 and 1 you could apply logistic regression.

-
Thanks! I have tried using "rlm" but the resultant QQ residual plot doesn't improve too much... I have updated my question... Thanks so much for your help! –  Luna Jul 6 '12 at 21:34