1
$\begingroup$

I am studying the 7th chapter of Introduction to Statistical Learning with Applications in R (ISLR) and I don't understand why the confidence interval associated with polynomial regressions (as seen in the first plot here) is so narrow.

According to the book (and to the R code provided), the dotted lines correspond to the predicted value, plus or minus twice the standard error - which, for normally distributed error terms, corresponds to ~95% confidence interval. However, according to the R code below (sorry it's so ugly), only 254 out of 3000 observations fall within the confidence interval:

library(ISLR)

age_range = range(Wage$age)
age.grid = seq(age_range[1], age_range[2])

fit = lm(wage~poly(age, 4), data=Wage)

preds = predict(fit, newdata=list(age=age.grid), se=TRUE)

within_interval = 0

for (i in 1:nrow(Wage)) {
    wage = Wage[i, "wage"]
    age = Wage[i, "age"]
    index = which(age.grid==age)
    pred = preds$fit[index]
    se = preds$se.fit[index]

    if (wage >= pred - 2*se & wage <= pred + 2*se) {
        within_interval = within_interval + 1
    }
}

So why does the model gives out such a narrow confidence interval? Are the errors not normally distributed, and is the confidence interval meaningless? (The residual plot does show a long right tail.) What am I missing here?

$\endgroup$
1
  • $\begingroup$ For a nice explanation of the meanings of the 95% confidence interval and prediction interval traces shown in a graph with a linear regression, see this two-minute video by Dr. Gerard Verschuuren: youtube.com/watch?v=XXa_xhCtZ-U $\endgroup$ – Dave Burton Oct 13 '16 at 13:30
1
$\begingroup$

Don't confuse a confidence interval, a quantile and a prediction interval.

Your confidence interval is a CI for the conditional mean, that is, for an unobserved and unobservable parameter of the underlying distribution. Here is its interpretation: if your model is correct and you repeat your experiment many times (sampling data, fitting the model and calculating a CI anew each time), then 95% of your CIs will contain the true underlying conditional mean.

However, your actual observations are distributed around this conditional mean, and the width of the observations' distribution is governed by the residual noise. Therefore, there is no reason to expect a given CI-for-the-mean to contain any specific percentage of the actual observations.

Look at it this way: if you simulate more and more data with a given conditional mean and noise and fit your model, then the CI for the mean will get narrower and narrower the more data you simulate, because a larger sample size means that the mean can be estimated better and better. However, if your noise level doesn't change, then this CI will contain fewer and fewer actual observations.

A similar confusion often appears in prediction. You can predict the future expected value and also give a confidence interval for it. But what people are usually interested in is not a CI for the future mean (which is, again, unobservable anyway), but a prediction interval that will contain a prespecified percentage of future observations. PIs will typically be larger than CIs.

You may be interested in the interval="prediction" argument to predict.lm().

$\endgroup$
1
  • $\begingroup$ Thanks a lot, Stephen. There is a lengthy previous discussion in the book of the difference between confidence and prediction intervals, and I completely forgot about it. $\endgroup$ – Lucas Murtinho Jan 7 '16 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.