1
$\begingroup$

Say I have some true values of some variable of interest $y_i^t$ for a population of individuals indexed by $i$. Now say I have some model-based predictions of those true values, which I'll denote by $y_i^p$. In my discipline, people often use the $R^2$ to assess how good the predictions are and I'm specifically interested in understanding how the $R^2$ is affected by the variance of $y_i^t$.

The standard approach is to run the following regression:

$$ y_i^t = \alpha + \beta y_i^p + \varepsilon_i. $$

We can then calculate the $R^2$ as follows:

$$ R^2 = 1 - \frac{SSR}{TSS} $$

where $SSR$ is the sum of squared residuals and $TSS$ is the total sum of squares. Now say I have a purely hypothetical prediction model that always achieves the same $SSR$ whether $y^t$ has a high or low variance. A higher-variance $y^t$ would then have a higher $TSS$, which would push up the $R^2$ given that $SSR$ is constant by assumption. That is, the $R^2$ is independently affected by the variance of $y^t$.

Now here's my point of confusion: $SSR$ is closely associated with the mean squared prediction error as $MSE = \frac{1}{N}SSR$. The MSE can be decomposed into parts related to the variance of the predictions, bias of the predictions, and some irreducible error. In the above thought experiment, the variance of the predictions must increase along with the variance of the true values to maintain a constant $SSE$ (or $MSE$). Does this imply that there is necessarily some compensating reduction in the bias or irreducible error in my thought experiment?

In short, given the $MSE$ decomposition, how can the $MSE$ remain constant while the variance in the predictions is increasing? Any help would be very much appreciated.

$\endgroup$

1 Answer 1

0
$\begingroup$

The $MSE\ (= \frac 1 N \sum_{i=1}^N(y_i^t-\hat y_i^t)^2)$ is independent of the variance of the test predictions ($\hat y_i^t$).

Are you confusing the model variance, which can be estimated using bias/variance decomposition, with the variance of the predictions over a test set from a single model?

When doing a bias/variance decomposition, we create multiple models (e.g. a set of regression models trained on different subsets of the training data). Lets say we create 20 such models. Then for each $y_i^t$ we have 20 estimates. It's the variance of these 20 estimates that is used in the bias/variance decomposition. We can aggregate the variances obtained for each $y_i^t$ to estimate the model variance and its contribution to the $MSE$, but this variance is not the same as the variance of the test predictions from a single model.

$\endgroup$
1
  • $\begingroup$ Yes, I was most definitely confusing model variance with the variance of the predictions from a single model. I figured I was misunderstanding something basic with the decomposition and that's it. I really appreciate your answer to my question. $\endgroup$ Commented Apr 21, 2022 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.