2
$\begingroup$
n <- 1000
betas <- c(1.2, 3, -2)
X <- cbind(var1 = rnorm(n, 0, 1), var2 = rnorm(n, 0, 2), var3 = rnorm(n, 0, 1.5))
set.seed(1)
epsilon <- rnorm(n, 0, 1)
y <-  X %*% betas + epsilon
data <- data.frame(cbind(y, X))
colnames(data) <- c("y", "var1", "var2", "var3")
> head(data)
          y        var1       var2       var3
1  9.983021  0.07730312  1.7000869 -2.7082253
2 -3.690355 -0.29686864 -1.8506260 -1.0170610
3  4.526042 -1.18324224  1.7871624 -0.7100371
4 -7.119478  0.01129269 -1.8820195  1.5411256
5  6.545304  0.99160104  1.0779042 -0.8960814
6 -3.479102  1.59396745 -0.3639488  1.7397741

I generate data from $Y = X\beta + \epsilon$. Assume that $X$ is fixed, and the only thing that is random is $\epsilon$. I divide the data set in half: half for training and half for testing.

# Split the data set into training and test set
train <- data[1:(n/2), ]
test <- data[(n/2 + 1): n, ]
test_X <- test[, names(test) != "y", drop = FALSE]

I fit an OLS model and calculate the empirical prediction error and get a value of 1.120582.

# Fit OLS model and calculate empirical prediction error
model <- lm(y ~ var1 + var2 + var3 - 1, data = train)
> mean((test$y - predict(model, newdata = test_X))^2)
[1] 1.120582

I then calculate the MSPE according to its formula:

\begin{align*} E||Y_{test} - X_{test} \hat{\beta}_{OLS}||_2^2 &= Var(X_{test}\hat{\beta}_{OLS}) + N_{test} * \sigma^2_{\epsilon}\\ &= tr\left(X_{test} Cov(\hat{\beta}_{OLS}) X_{test}^T\right) + N_{test} * \sigma^2_{\epsilon} \end{align*}

In this case, $N = 1000,$ so $N_{test} = N/2 = 500$, and $\sigma^2_{\epsilon} = 1$. Using the above formula, I get an MSPE of 503.0804, which is very different from the 1.120582 I found above. I understand that MSPE is an expected value, i.e., ideally I would generate many data sets and average across their prediction error. However, the difference between 503.0804 and 1.120582 is so big that it leads me to think that I might have written down the formula or coded it wrong. Any thoughts?

# Calculate the MPSE 
> sum(diag(as.matrix(test_X) %*% as.matrix(vcov(model)) %*% as.matrix(t(test_X)))) + nrow(test_X) * sigma2
[1] 503.0804
$\endgroup$

1 Answer 1

1
$\begingroup$

Good question. Let $(y,x)$ be your test data with $y = x^T \beta^* + \varepsilon$ and assume that $x$ is random as well (independent of $\varepsilon$). To simplify, let $$\delta = \beta^* - \widehat \beta.$$ Then, your mean-squared prediction error is \begin{align*} \mathbb E [(y - x^T \widehat \beta)^2 \mid \mathcal T] &= \mathbb E [(x^T \beta^* - x^T \widehat \beta + \varepsilon)^2 \mid \mathcal T] \\ &= \mathbb E[(x^T\delta ^2 + \varepsilon^2 \mid \mathcal T] \\ &= \delta^T \Sigma_x \delta + \sigma^2 \end{align*} where $\mathcal T$ is the training data and $\Sigma_x$ is the covariance of $x$ assuming it is zero mean, i.e., $\Sigma_x = \mathbb E xx^T$.

In your example, $\sigma^2 = 1$ and $$ \Sigma_x = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 2^2 & 0 \\ 0 & 0 & 1.5^2 \end{pmatrix}. $$

Note that covariance of $\widehat\beta$ does not come into play.


What do you actually compute in practice via the training and test splits? Suppose you have $y_i = x_i^T \beta^* + \varepsilon_i$ and let $y = (y_1,\dots,y_n)$ and $X$ have rows $x_i^T$. I am assuming that these are all test data, i.e., $(y,X)$ is the test data.

What does the following line of code compute?

mean((test$y - predict(model, newdata = test_X))^2)

This code only averages over the test data. It computes the following \begin{align*} \frac1n \| y - X \widehat \beta\|^2 &= \frac1n \| X \beta^* - X \widehat \beta + \varepsilon\|^2 \\ &= \frac1n \| X \delta + \varepsilon\|^2 \\ &= \delta^T \frac{X^T X}{n} \delta + 2 \delta^T \Big(\frac{1}{n} X^T \varepsilon \Big) + \frac{\|\varepsilon\|^2}{n}. \end{align*} Let $$\widehat \Sigma_x = \frac{1}{n} X^T X, \quad \text{ and } \widehat \Sigma_{x \varepsilon} = \frac{1}{n} X^T \varepsilon.$$ Then, \begin{align*} \frac1n \| y - X \widehat \beta\|^2 &= \delta^T \widehat \Sigma_x \delta + 2 \delta^T \widehat \Sigma_{x\varepsilon} + \frac{\|\varepsilon\|^2}{n}. \end{align*} If you would like to assume $x_i$ are nonrandom, you can for example, define your prediction error as $$ \frac1n \mathbb E \| y - X \widehat \beta\|^2 = \delta^T \widehat \Sigma_x \delta + \sigma^2 $$ since $\mathbb E [X^T \varepsilon] = X^T \mathbb E [\varepsilon]$. In the above argument the expectation is w.r.t. to the test data (conditional on the training effectively.) The only difference here is that you are using the sample covariance matrix of the test data $\widehat \Sigma_x$ instead of the population level covariance $\Sigma_x$.

Depending on what you are willing to assume fixed versus random, you get different approximations, form exact (i.e., both $X$ and $y$ of test are considered fixed) to the more accurate one where only $X$ is fixed to the less accurate one where you take the expectation over both $x$ and $y$.


But what about the slide 4 linked in the comments? Covariance of $\widehat \beta$ does not come into play, unless you want to repeat the training many many times and take an average over those as well (but this is not what your code is doing ... you are only training once). In case you repeat the experiment by averaging over many training/test splits then you can evaluate $$ \mathbb E (\delta^T \widehat \Sigma_x \delta ) = \mathbb E (\text{tr}( \widehat \Sigma_x \delta \delta^T)) = \text{tr}( \widehat \Sigma_x \mathbb E \delta \delta^T) = \text{tr}( \widehat \Sigma_x \text{cov}(\widehat \beta)). $$ where $\mathbb E$ is now over the training data.

$\endgroup$
7
  • $\begingroup$ What if I assume $X$ is fixed? How might that change the formula? Also, $\epsilon$ here is a vector, so should it be $\mathbb{E}(\epsilon^2) = \mathbb{E}\left(\sum_{i=1}^{N_{test}} \epsilon_i^2\right) = N_{test} * \sigma^2_{\epsilon}$ and not $\mathbb{E}(\epsilon^2) = \sigma^2_{\epsilon}$? $\endgroup$
    – Adrian
    Commented Jan 21, 2021 at 2:16
  • $\begingroup$ The equation I found is on slide 4 of this set of slides: dept.stat.lsa.umich.edu/~kshedden/Courses/Regression_Notes/… $\endgroup$
    – Adrian
    Commented Jan 21, 2021 at 2:18
  • $\begingroup$ @Adrian, added details for your what if scenario. $\endgroup$
    – passerby51
    Commented Jan 21, 2021 at 2:43
  • 1
    $\begingroup$ thanks for such a detailed explanation. I should've added more code to clarify what I was actually doing. I actually want to follow slide 4 linked in the comments and repeat training many many times. Thank you again. $\endgroup$
    – Adrian
    Commented Jan 21, 2021 at 2:58
  • 1
    $\begingroup$ @Adrian, yes something like that. Or you can replace mean with sum when you compute it empirically and avoid dividing by nrow(test_X). Though, I prefer the mean myself. $\endgroup$
    – passerby51
    Commented Jan 21, 2021 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.