Does anyone know of a metric that quantifies the bias-variance tradeoff of a given fitted model?

I'm not talking about measuring the MSE in cross validation, I'm interested in a single generic or model-specific metric (or statistical test) that measures the degree of bias and/or variance of a fitted model.

• Knowing the bias would mean knowing the truth, in which case you would not need a model. You can never measure the bias without knowing the truth. Commented Apr 9, 2017 at 5:36
• This seems like a perfectly good, & clear, question to me. I'd be interested in reading some good answers, too. I'm voting to leave open. Commented Apr 9, 2017 at 11:58
• @MatthewDrury considering that the answer talks about the MSE as well, I think the OP may have data on the truth, like in regression analysis. In many cases you may still want to model it for forecasting purposes Commented Jul 25, 2017 at 11:40

$$EEPSE = EE(\hat{f}(x_0)-y_0)^2$$ is the average (or "expected", over different predictions $$\hat{f}(x_0)$$ coming from different datasets) expected prediction squared error (for a particular $$\hat{f}(x_0)$$ over many test points $$y_0$$, at $$x_0$$)

To measure the bias-variance trade-off in a concrete example with concrete values, consider e.g. a population of 3 members: $$0$$, $$0.5$$, and $$1$$, each one occurring with the same probability. (The population is infinite). Now consider random datasets that consist of one observation from the population. There will be 3 types of datasets, which occur with the same (1/3) frequency. These datasets have observations "$$0$$", "$$0.5$$", and "$$1$$". Now consider models of type: "The prediction for the next observation is equal to the average in the dataset times alpha". The average in the dataset is equal to the observation in the dataset, as we have just one observation in a dataset.

To measure the bias-variance tradeoff among different models (one model per "alpha"), plot the squared bias $$[E(\hat{f}(x_0))-\mu_0]^2$$, the variance $$E(\hat{f}(x_0) - E(\hat{f}(x_0)))^2$$, their sum, and their sum plus noise (which is the $$EE(\hat{f}(x_0)-y_0)^2$$). The "noise" is equal to $$var(y_0)$$.

What you will get is that the optimal alpha is $$0.6$$. Also, you will see that the unbiased model (alpha = $$1$$) can never be optimal, as the slope of the EEPSE is positive for alpha = $$1$$ (see the figure). This picture is true in general. There does not exist an unbiased model, which is optimal. The only two exceptions are: the dataset is infinite (and equal to the population), or the population consists of one member only. In these two corner cases (not practical) alpha=$$1$$ is optimal, as then $$var(\hat{f}(x_0)) = 0$$.

The optimal alpha is $$\frac{\mu_0^2}{\mu_0^2 + var(\hat{f}(x_0))} = \frac{0.5^2}{0.5^2 + var([0, 0.5, 1])} = 0.6,$$ which is always between $$0$$ and $$1$$, where $$\hat{f}(x_0)$$ is the prediction from the unbiased model "The prediction for the next observation is equal to the average in the dataset". The optimal model is:

The prediction for the next observation is equal to the average in the dataset times 0.6

. This result may look "strange". It means that if we have one dataset, and it consists of (the single) observation "1", our prediction for the future observations should be 1 $$\times$$ alpha = 0.6; and if we have as observation "0.5", our prediction should be 0.5 $$\times$$ 0.6 = 0.3. Lastly, if we have as observation "0", our prediction should be 0 $$\times$$ 0.6 = 0. This model results in a smaller $$EEPSE$$ than the model "the prediction for the next observation is equal to the average in the dataset". Note that the "average in the population" results in the smallest $$EEPSE$$ (0.166667). So, we would like to predict "the average in the population". Alas, the "average in the dataset" is not the best prediction for the "average in the population".

Multiplying an unbiased model by alpha in this case is called penalization. Note that the optimal alpha is in general unknow, as one needs to know the true $$\mu_0$$ in order to compute it. Still, the point to stress is that optimal alpha < $$1$$. Even if a small penalization is used (alpha very close to $$1$$), we will still have smaller test error than the unbiased model.

$E(y_0-\hat{f}(x_0))^2 = Var(\hat{f}(x_0)) + Bias(\hat{f}(x_0))^2 + Var(\epsilon)$

Where
- $E(y_0-\hat{f}(x_0))^2$ is the expected test MSE.
- $Var(\hat{f}(x_0))$ is variance of fitted model.
- $Bias(\hat{f}(x_0))^2$ is squared biased of model.
- $Var(\epsilon)$ is variance of error terms

Source: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112). New York: springer.

• It was the same paragraph from the Elements of statistical learning that made me ask this question. I was however asking about specialised metrics that would tell you specifically how to derive Bias(.) rather than assume it. Commented Jul 25, 2017 at 10:53