Does anyone know of a metric that quantifies the bias-variance tradeoff of a given fitted model?

I'm not talking about measuring the MSE in cross validation, I'm interested in a single generic or model-specific metric (or statistical test) that measures the degree of bias and/or variance of a fitted model.

  • 2
    $\begingroup$ Knowing the bias would mean knowing the truth, in which case you would not need a model. You can never measure the bias without knowing the truth. $\endgroup$ – Matthew Drury Apr 9 '17 at 5:36
  • 1
    $\begingroup$ This seems like a perfectly good, & clear, question to me. I'd be interested in reading some good answers, too. I'm voting to leave open. $\endgroup$ – gung Apr 9 '17 at 11:58
  • 1
    $\begingroup$ @MatthewDrury considering that the answer talks about the MSE as well, I think the OP may have data on the truth, like in regression analysis. In many cases you may still want to model it for forecasting purposes $\endgroup$ – KenHBS Jul 25 '17 at 11:40

$E(y_0-\hat{f}(x_0))^2 = Var(\hat{f}(x_0)) + Bias(\hat{f}(x_0))^2 + Var(\epsilon)$

- $E(y_0-\hat{f}(x_0))^2$ is the expected test MSE.
- $Var(\hat{f}(x_0))$ is variance of fitted model.
- $Bias(\hat{f}(x_0))^2$ is squared biased of model.
- $Var(\epsilon)$ is variance of error terms

Source: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112). New York: springer.

  • $\begingroup$ It was the same paragraph from the Elements of statistical learning that made me ask this question. I was however asking about specialised metrics that would tell you specifically how to derive Bias(.) rather than assume it. $\endgroup$ – Digio Jul 25 '17 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.