Why do we estimate the mean response in Confidence interval but predict individual outcome in prediction? I have tried to understand the differences. Just to be clear, I think I understand the two sources of variation in prediction comes from the variation in the distribution of the location of Y and within variation of Y. The thing I do not get is, why we do not use the estimate of the mean response (like in estimation) but use the individual outcome of Y (thus the two variation) in prediction? Or to put it other way round, why in estimation we did not consider variation in the distribution of the location of Y? Is it because, Y is a rv in prediction? If so, why Y is not rv in estimation?
 A: The difference between the confidence interval for the mean response and the prediction interval is subtle but important. I'll explain it first and then provide you with a graphical intuition which helped me a lot when I learned this.
Obviously, there is an error component to our prediction. Under the normal probability model, the prediction is normally distributed about the mean response. The problem is that the mean response is a random variable that also has an error component associated to it. Given a sample of size $N$ we will have an estimate for the mean response: the confidence interval describes where we would expect to find the mean response if we had built our model using different samples of size $N$ from our population. So there is an upper and lower bound to where we expect to find the mean response, and the distribution of our predictions could be centered anywhere between these bounds. So the prediction interval contains the confidence interval for the mean response, with a tail added to each end of the interval to encompass the error we would expect for a prediction if the mean response were fixed at that location (which it's not).
Let's make this concrete with an example. Let's say for some level of $X$, our model predicts a mean response of 0 with unit variance (i.e. the mean response is distributed according to the standard normal).

A given preduction is distributed with mean equal to the mean response, so if we're only considering values within a 95% confidence interval, the distribution of our prediction could be centered as far left as the lower limit of the CI or as far right as the upper limit of the CI.

We can then intuit graphically that the prediction interval is given by the lower limit given when we anticipate the mean response will be located at the lower extreme and the upper limit found when we anticipate the mean response is located at the upper extreme.

This example was inspired by Figure 2.5 (p. 58) of Applied Linear Statistical Models by Kutner, Nachsteim, Neter and Li.
Code used for this example, for posterity:
xv=seq(from=-5,to=5, length.out=1e4)
plot(xv, dnorm(xv), type='l', xlim=c(-5,5), main="95% CI for the Mean Response")
abline(v=0)
abline(v=qnorm(.975), lty=2)
abline(v=-qnorm(.975), lty=2)


plot(xv, dnorm(xv), xlim=c(-5,5), type='n', main="Distribution of predictions given mean \nresponse is located at extremes of CI")
abline(v=qnorm(.975), col="blue")
abline(v=-qnorm(.975), col="blue")
abline(v=0, col="blue", lty=2)
lines(xv, dnorm(xv, qnorm(.975)), col='blue')
lines(xv, dnorm(xv, -qnorm(.975)), col='blue')
abline(v=2*(-qnorm(.975)), lty=2, col='blue')
abline(v=2*(qnorm(.975)), lty=2, col='blue')

plot(xv, dnorm(xv), xlim=c(-5,5), type='l', main="95% CI vs. 95% Prediction Interval")
lines(xv, dnorm(xv,0,2), col="blue")
abline(v=0)
abline(v=qnorm(.975), lty=2)
abline(v=-qnorm(.975), lty=2)
abline(v=2*(-qnorm(.975)), lty=2, col='blue')
abline(v=2*(qnorm(.975)), lty=2, col='blue')

A: Here is an attempt to answer my own question. 
Let me try to answer this question in terms of "simple regression". 
In regression, you have observations (X) and response (Y).Each Y is assumed to have a normal distribution and you want to estimate the mean of this Y, i.e. E(Y|x), sometimes also referred to as mean response. 
Say, Y_hat gives this estimate using Least square methods ( that is, betas are estimated using least squares). Now, we know that we only have one sample, that is, for a different sample but a same level of X, you may have different Y_hat. So there is a inherent variance in  in Y_hat from sample to sample. This variance is known as Standard Error, which takes into account that we have only one sample.Thus, the confidence interval uses this standard error. Whereas,
now, we use the same equation to predict a new observation X(new). The difference between these two lies in that, in confidence interval we estimated mean response E(Y|x), which is a parameter. In prediction, however, we are trying to estimate (predict) an actual (individual) value of Y, which is a random variable. When doing this, there are two sources of variation. The first variation  takes into account the variation in Y|x (which can sometimes be viewed as location and has a distribution) and the second variation is the within variation (think of it as once the location has been determined), which we already calculated when constructing CI, that is, this captures the variation in mean response E(y|x), the regression function. 
So, this is why the prediction interval are wider than the CI as they account for two variability. This calculation of variation is the only difference between CI and PI. 
So, three important to keep in mind is:


*

*For each Y, there is a distribution for each level of X ( an assumption of regression). 

*In CI, we estimate E(Y|x) and build interval around it and it is a parameter. 

*In PI, we estimate actual Y|x and it is a random variable. 

A: Good question with an important difference. The Confidence Interval and the Prediction interval represent two different ideas.
The Confidence Interval explains how much variation could be possible with a certain model. After a regression, given random normal errors, the real regression line could be different from what we estimate. This is calculated based on the SE, and is represented by a confidence interval. Essentially, we are giving uncertainty for $E[Y]$. Thus, it is a band around our estimation that gives an idea of the certainty of the model. If the bands are large (large SE), then the model could well be useless! 
The Prediction Interval aims to point out the uncertainty in $E[Y|X]$. That is to say, we have measured all the values in our model and predicted the outcome, $Y$. What is the uncertainty in this one specific estimate? Logically, the error should be greater than the confidence interval. The calculation is based on the quantiles of the estimate of the variance of SE.
To make a link between the two. If we take 100 samples from the same point (100 samples of $E[Y|X]$, we expect them to be distributed mostly within the prediction interval. However, we expect their mean to be within the confidence interval. 
