Does the non-normality matter in using regression for prediction?
In the QQ plot of the residuals after linear regression, the residuals turned out to be highly non-Gaussian.
Most of the points (95%) are below the 45 degree straightline. And those below the straightline are all on the lower side.
The shape of the curve looks like f(x)=x^(1/5) for x on [0, 1]. (The 5% points in the middle of this curve are above the 45 degree straight line).
It turns out that the dependent variable y has data that are highly non-Gaussian. They are all in [0, 1], but mostly clustered around 1.
So I tried various ways of transforming y. The latest one I've found was to do y_new = y ^ 7 .
Using this transformation, the y_new data become much more symmetrical than before but they are still no way look like Gaussian in histogram.
Being very disappointed, I don't have any more weapon in my bag...
Instead, I began to wonder, if my end goal is just to get the yhat, i.e. the prediction of the data on a wider data-set,
does the Non-normality even matter, in terms of accuracy for prediction?
Please shed some lights on me.
The y data looks very similar to the data generated using the following script:
mydata=rt(10000, df=5) mydata=mydata[mydata<0.8] mydata=(mydata-min(mydata))/(max(mydata)-min(mydata)) hist(mydata, 100)