3
$\begingroup$

Does the non-normality matter in using regression for prediction?

Hi all,

In the Q-Q plot of the residuals after linear regression, the residuals turned out to be highly non-Gaussian. Most of the points (95%) are below a 45º straight line. And those below the straight line are all on the lower side.

The shape of the curve looks like $f(x)=x^{1/5}$ for $x$ on $[0, 1]$. (The 5% points in the middle of this curve are above the 45º straight line).

It turns out that the dependent variable $y$ has data that are highly non-Gaussian. They are all in $[0, 1]$, but mostly clustered around $1$.

So I tried various ways of transforming $y$. The latest one I've found was to do $y_{new} = y^7$.

Using this transformation, the $y_{new}$ data become much more symmetrical than before, but they still in no way look Gaussian in a histogram.

Being very disappointed, I don't have any more weapons in my bag...Instead, I began to wonder, if my end goal is just to get the $\hat{y}$, i.e. the prediction of the data on a wider data-set, does the non-normality even matter, in terms of accuracy for prediction?

Please shed some light for me. Thank you!

The $y$ data looks very similar to the data generated using the following script:

mydata=rt(10000, df=5)
mydata=mydata[mydata<0.8]
mydata=(mydata-min(mydata))/(max(mydata)-min(mydata))

hist(mydata, 100)
Improve this question
$\endgroup$
8
  • $\begingroup$ If you could provide more information about y maybe we could be of more assistance. In short tailed cases the nonnormality is not as severe a problem as in heavytailed case. Have you tested your regression model on data that wasn't used in the fitting process? $\endgroup$
    – Michael R. Chernick
    Jul 6, 2012 at 20:59
  • $\begingroup$ Yes, I've tried that. I have also updated my question, hopefully it will answer your question better. Thanks a lot! $\endgroup$
    – Luna
    Jul 6, 2012 at 21:27
  • $\begingroup$ I disagree with AdamO and Greg Snow on the general statement that nonnormality does not hurt prediction. For heavytailed distributions at least, least squares is overly sensitive to outliers and leverage points. This affects the accuracy of the regression coefficients and consequently your prediction accuracy. Assume the form of the model is correct and the residuals have zero mean and constant variance but are not normal then the Gauss-Markov theorem says that they will still be minimum variance among unbiased estimators. In that sense the estimates are still good. $\endgroup$
    – Michael R. Chernick
    Jul 6, 2012 at 21:34
  • $\begingroup$ I think that is the motivation that AdamO and Greg Snow to say that they are still good for predicting new values of y. But the best predictor need not always be unbiased. Robust regression can produce better estimates when the distribution of the residuals are heavytailed. But this does not seem to be you problem. $\endgroup$
    – Michael R. Chernick
    Jul 6, 2012 at 21:39
  • $\begingroup$ Thanks Mike. Is my data heavy-tailed or not? Thank you! $\endgroup$
    – Luna
    Jul 6, 2012 at 23:06

3 Answers 3

Reset to default
5
$\begingroup$

Normality is not needed to fit the regression line, the normality is used when making inference about it (hypothesis tests and confidence intervals). Without normality the regression equation will still predict the mean (assuming the correct model), but a prediction interval based on normality and that mean will also be meaningless. However, in highly skewed cases the mean may not be the most meaningful statistic, which is why things like robust regression, quantile regression, and others are often suggested. With a y value between 0 and 1 you might consider Beta regression.

Most of this comes down to figuring out what question you are trying to answer and what makes sense relative to the question and the science (does $y^7$ have any scientific meaning? if not I would avoid using it).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks a lot! I have updated my question with some code to show what the y data look like. In theory, the y data may not be always in [0, 1]. However, in reality, I found that min(y) = -0.06393158, max(y)=0.9942669... can I still use Beta regression? What's the benefit of using Beta regression vs. Linear regression? More accurate? Thank you! $\endgroup$
    – Luna
    Jul 6, 2012 at 21:32
  • $\begingroup$ Beta regression assumes that the response comes from a Beta distribution (like gaussian regression assumes a normal). So it is only for data between 0 and 1 but works well for things like proportions when you don't have the denominator to do logistic regression. $\endgroup$
    – Greg Snow
    Jul 6, 2012 at 21:50
  • $\begingroup$ @GregSnow could you maybe provide some reference? $\endgroup$
    – user1607
    Jun 4, 2019 at 19:55
  • $\begingroup$ @user1607, See: cran.r-project.org/web/packages/betareg/betareg.pdf for the R package to do Beta Regression. Page 8 of that document has references to other resources and background. $\endgroup$
    – Greg Snow
    Jun 5, 2019 at 15:05
2
$\begingroup$

As I told you with your previous question the prediction can be highly influenced by nonnormality and robust regression is an alternative. It sounds like the nonnormality is due to very short tails and heavy skewness. You say that your DV is confined to [0,1] with high concentration at 1. That certainly is short tailed and it would be skewed if there are a few point far away from 1. If y were just a binary response only taking the values 0 and 1 you could apply logistic regression.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks! I have tried using "rlm" but the resultant QQ residual plot doesn't improve too much... I have updated my question... Thanks so much for your help! $\endgroup$
    – Luna
    Jul 6, 2012 at 21:34
1
$\begingroup$

If you're using MSE as the usual measure of predictive accuracy in the data, then departures from normality are not going to hurt your predictions at all. A consideration, though, is whether high leverage / high influence points are affecting your estimates. Usually, a scatter plot of fitted values versus residuals can show this as residuals will not appear evenly spread around the 0 horizontal line.

Another consideration is whether the linearity assumption doesn't hold and whether a slightly more advanced non-linear method such as smoothing splines or loess curves would give you better predictions. In using the loess, it will throw out highly influential points.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks! Agreed that it seems that a linear regression won't fit well; however, a smoothing splines will fit too well... to the extent of over-fitting... Are there something in between? Thank you! $\endgroup$
    – Luna
    Jul 6, 2012 at 21:33
  • 1
    $\begingroup$ Loess is a nonparametric smoothing function. I don't know what AdamO is talking about when he says that loess throws out influential observations. It gives weight to all the data in a neighborhood of a given y. $\endgroup$
    – Michael R. Chernick
    Jul 7, 2012 at 0:21
  • $\begingroup$ @Luna Can you explain why the dependent variable is concentrated on [0, 1] and mostly concentrated near 1? $\endgroup$
    – Michael R. Chernick
    Jul 7, 2012 at 0:22
  • $\begingroup$ That's because the experiment measurement results gave these numbers... $\endgroup$
    – Luna
    Jul 9, 2012 at 15:47
  • $\begingroup$ @MichaelChernick You're right, I meant to say high leverage points. The loess using the tricubic weighting function which places zero weights on points beyond a local neighborhood. $\endgroup$
    – AdamO
    Jul 10, 2012 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.