Prediction interval with non normal data (In the context of simple or multiple linear regression)
Perhaps a very simple question to most of you. I know that if your sample size is large enough, despite your data not following a normal distribution, it would be viable to still compute the confidence interval due to the Central Limit Theorem.
What I heard is that this is not the case for a prediction interval. Would someone be able to explain me in plain English what the reason for this is? And if possible provide some mathematics after? :) 
Much appreciated!
 A: Recall the differences between prediction and confidence intervals: a confidence interval is an interval for a mean which converges to a fixed value as the sample size goes to infinity. The prediction interval is an interval for an observation which goes to a non-singular interval as the sample size goes to infinity.
Since the confidence interval applies to a mean, the CLT gives us a normal approximate sampling distribution for that sample mean. If you incorrectly apply a normal approximation to a prediction interval, you do not account for the heteroscedasticity, skewness, or the kurtosis of the distribution, or even the scale of the response levels. For instance, if I have binary data, it makes sense to create a normal interval for the sample proportion. But no amount of correction will give me a sensible "prediction" interval for a binary valued response: you would (instead) report the probability $Y=1$ or $0$. 
A: Consider a very simple case -- constructing a confidence interval for a common population mean compared to constructing a prediction interval for a single new observation.
With the assumption of independence and a common distribution with finite variance ($Y_i\sim F$, with $E(Y_i)=\mu,\text{Var}(Y_i)=\sigma^2$), then in the limit as $n\to\infty$, $\frac{\bar{Y}-\mu}{s/\sqrt{n}}$ will converge to a standard normal random variate, and from that we can back out an approximate large-sample confidence interval for $\mu$ based on a sample.
In particular, for a certain skewed distribution, here's a simulated distribution of sample means for $n=50$ (NB there's nothing special about $n=50$, or any other specific sample size):

We see that in this case the sample means appear not very far from normal even at n=50 (still slightly skew though, as the original distribution was pretty skewed). In cases similar to this one, "large sample" methods for a confidence interval may work reasonably well at sample sizes of a few dozen. In other cases it might require substantially larger samples.
However, even if we knew $\mu$ exactly, we won't know from results like the above where $Y_{n+1}$ might be. We can't make use of such arguments as we used for the mean because we are dealing with the distribution of a single observation. We can still construct a pivotal-type quantity like $\frac{Y_{n+1}-\bar{Y}}{s/\sqrt{n}}$, but there's no large sample result to invoke here.

[That's not to suggest nothing whatever can be done to try to construct some form of interval for a future observation, just that we can't invoke large-sample results relating to convergence of standardized means to normality to do it. If you have a distributional model for $F$, you may be able to construct a sensible interval for a future observation. Or you may be able to construct some form of nonparametric interval in some situations]
Somewhat similar issues hold for say a regression model. 
