I'm currently trying to complete the book 'Introduction to Statistical Learning' by James et al. and I'm stuck in one of the exercises, trying to get some logic out of the results.
The question is this one (see in line):
- I collect a set of data (n = $100$ observations) containing a single predictor and a quantitative response. I then fit a linear regression model to the data, as well as a separate cubic regression, i.e. $Y = β_0 +β_1X +β_2X^2 +β_3X^3 +\varepsilon$.
(a) Suppose that the true relationship between X and Y is linear, i.e. $Y = β_0 + β_1X + \varepsilon$. Consider the training residual sum of squares (RSS) for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.
Given that the true relationship is linear, I thought adding a cubic term wouldn't improve the fit, it would take up degrees of freedom, and simple would increase the RSS. I tried to test this empirically:
training <- function(p, n) {
set.seed(1)
x <- rnorm(n)
y <- 5 + 2*x + rnorm(n, sd = 0.5)
sigma_linear <- summary(lm(y[p] ~ x[p]))$sigma
sigma_polynomial <- summary(lm(y[p] ~ x[p] + I(x[p]^2) + I(x[p]^3)))$sigma
c(linear = sigma_linear, polynomial = sigma_polynomial)
}
training(1:20, 100)
# linear polynomial
# 0.4177607 0.4203941
If you run this, the linear RSS is smaller than the cubic one, which is what I would expect. But remove the set.seed option from the function and replicate this a couple of times and in many cases the cubic one will be higher. This might be simple random noise and the cubic term doesn't add or remove anything from the model. I don't understand why the cubic model would give a better 'fit'. In fact, in that exact link, both higher power terms are insignificant and the RSE is higher for the cubic model.
(b) Answer (a) using test rather than training RSS.
Now, using the remaining part of the data:
training(20:100, 100)
# linear polynomial
# 0.4957765 0.4984665
The cubic sigma is still higher. But again, replicate without the set.seed option and it will sometimes be smaller or higher.
Why do then people expect for the first cubic model to have a lower RSS in the linear model?