Test and Train RSS in OLS model

Question

I encountered the following true/false question:

Given a train sample with $\ N $ observations and OLS model fitted on that sample, the RSS of the train sample will be less than or equal to the expectancy of the RSS of all test samples from the same distribution and in the same sample size.

The answer is false. I find it hard to understand why. I think I can understand why it may be the case that for some test sample, the RSS will be lower than the train RSS, but how could it be that the expectancy of all test samples will be lower than the train RSS?

Answer 1

In general, "the RSS of the train sample" is a random variable, while "expectancy of the RSS of all test samples from the same distribution and in the same sample size" is a fixed number, so actually they are not directly comparable (and because of that, there is positive probability that the former one is greater than the latter).

A very similar, but more meaningful comparison is Exercise 2.9 in The Elements of Statistical Learning:

Consider a linear regression model with $p$ parameters, fit by least squares to a set of training data $(x_1, y_1), \ldots, (x_N, y_N)$ drawn at random from a population. Let $\hat{\beta}$ be the least squares estimate. Suppose we have some test data $(\tilde{x}_1, \tilde{y}_1), \ldots, (\tilde{x}_M, \tilde{y}_M)$ drawn at random from the same population as the training data. If $R_{\text{tr}}(\beta) = \frac{1}{N}\sum_{1}^N(y_i - \beta^Tx_i)^2$ and $R_{\text{te}}(\beta) = M^{-1}\sum_{1}^M (\tilde{y}_i - \beta^T\tilde{x}_i)^2$, prove that \begin{equation*} E[R_{\text{tr}}(\hat{\beta})] \leq E[R_{\text{te}}(\hat{\beta})], \end{equation*} where the expectations are over all that is random in each expression.

With the notations in the quotation block, as opposed to the correct assertion $E[R_{\text{tr}}(\hat{\beta})] \leq E[R_{\text{te}}(\hat{\beta})]$, your conjecture in the OP is $R_{\text{tr}}(\hat{\beta}) \leq E[R_{\text{te}}(\hat{\beta})]$, which is not true as explained in the first paragraph.

Answer 2

Here is an example of that very behavior.

library(ggplot2)
set.seed(2024)
N <- 100
R <- 10000
x <- sort(runif(N, -2, 2))
e <- c(17, rnorm(N - 2, 0, 1), -17)
Ey <- 2 - x
y <- Ey + e
 
L <- lm(y ~ x)
rss_in <- sum((y - predict(L))^2)
 
rss_out <- rep(NA, R)
for (i in 1:R){
 
  e <- rnorm(N, 0, 1)
  y <- Ey + e
  preds <- predict(L)
  resids <- y - preds
  rss_out[i] <- sum((resids)^2)
}
plot(density(rss_out))
abline(v = rss_in)
 
d <- data.frame(
  RSS_Out = rss_out,
  C = "Out-of-sample Residual Sum of Squares"
)
ggplot(d, aes(x = RSS_Out, fill = C)) +
  geom_density(alpha = 0.2) +
  # geom_vline(xintercept = rss_in) +
  theme(legend.position = "none")
range(rss_out)

In this example, the out-of-sample residual sum of squares ranges from $75$ to $209$, so the expected value is (probably) in that range. However, the in-sample residual sum of squares is much higher at $641$.

The example works on the idea that, sure, the expected value of the out-of-sample residual sum of squares uses the same standard normal error term as there is in-sample, except that, for this particular set of in-sample data, there was some rotten luck where that standard normal error threw two huge errors.

If you are formal about showing what the expected out-of-sample residual sum of squares is and that it is lower than $641$, then you have a complete proof. I am content to let these particular $x$ values be the only possible values. If you want to change $x$ to be binary to make that assumption more realistic, x <- c(rep(0, N/2), rep(1, N/2)) tells the same story with slightly different numbers.