Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Does anyone know of a metric that quantifies the bias-variance tradeoff of a given fitted model?
I'm not talking about measuring the MSE in cross validation, I'm interested in a single generic or model-specific metric (or statistical test) that measures the degree of bias and/or variance of a fitted model.
$EEPSE = EE(\hat{f}(x_0)-y_0)^2$ is the average (or "expected", over different predictions $\hat{f}(x_0)$ coming from different datasets) expected prediction squared error (for a particular $\hat{f}(x_0)$ over many test points $y_0$, at $x_0$)
To measure the bias-variance trade-off in a concrete example with concrete values, consider e.g. a population of 3 members: $0$, $0.5$, and $1$, each one occurring with the same probability. (The population is infinite). Now consider random datasets that consist of one observation from the population. There will be 3 types of datasets, which occur with the same (1/3) frequency. These datasets have observations "$0$", "$0.5$", and "$1$". Now consider models of type: "The prediction for the next observation is equal to the average in the dataset times alpha". The average in the dataset is equal to the observation in the dataset, as we have just one observation in a dataset.
To measure the bias-variance tradeoff among different models (one model per "alpha"), plot the squared bias $[E(\hat{f}(x_0))-\mu_0]^2$, the variance $E(\hat{f}(x_0) - E(\hat{f}(x_0)))^2$, their sum, and their sum plus noise (which is the $EE(\hat{f}(x_0)-y_0)^2$). The "noise" is equal to $var(y_0)$.
What you will get is that the optimal alpha is $0.6$. Also, you will see that the unbiased model (alpha = $1$) can never be optimal, as the slope of the EEPSE is positive for alpha = $1$ (see the figure). This picture is true in general. There does not exist an unbiased model, which is optimal. The only two exceptions are: the dataset is infinite (and equal to the population), or the population consists of one member only. In these two corner cases (not practical) alpha=$1$ is optimal, as then $var(\hat{f}(x_0)) = 0$.
The optimal alpha is $$\frac{\mu_0^2}{\mu_0^2 + var(\hat{f}(x_0))} = \frac{0.5^2}{0.5^2 + var([0, 0.5, 1])} = 0.6, $$ which is always between $0$ and $1$, where $\hat{f}(x_0)$ is the prediction from the unbiased model "The prediction for the next observation is equal to the average in the dataset". The optimal model is:
The prediction for the next observation is equal to the average in the dataset times 0.6
. This result may look "strange". It means that if we have one dataset, and it consists of (the single) observation "1", our prediction for the future observations should be 1 $\times$ alpha = 0.6; and if we have as observation "0.5", our prediction should be 0.5 $\times$ 0.6 = 0.3. Lastly, if we have as observation "0", our prediction should be 0 $\times$ 0.6 = 0. This model results in a smaller $EEPSE$ than the model "the prediction for the next observation is equal to the average in the dataset". Note that the "average in the population" results in the smallest $EEPSE$ (0.166667). So, we would like to predict "the average in the population". Alas, the "average in the dataset" is not the best prediction for the "average in the population".
Multiplying an unbiased model by alpha in this case is called penalization. Note that the optimal alpha is in general unknow, as one needs to know the true $\mu_0$ in order to compute it. Still, the point to stress is that optimal alpha < $1$. Even if a small penalization is used (alpha very close to $1$), we will still have smaller test error than the unbiased model.
$E(y_0-\hat{f}(x_0))^2 = Var(\hat{f}(x_0)) + Bias(\hat{f}(x_0))^2 + Var(\epsilon)$
Where
- $E(y_0-\hat{f}(x_0))^2$ is the expected test MSE.
- $Var(\hat{f}(x_0))$ is variance of fitted model.
- $Bias(\hat{f}(x_0))^2$ is squared biased of model.
- $Var(\epsilon)$ is variance of error terms
Source: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112). New York: springer.
