3
$\begingroup$

Let $y_i$ be the true value and $\hat{y}_i$ a prediction from a model. Then, for example $$B=n^{-1}\sum_{i=1}^n \hat{y}_i - y_i$$ is the prediction bias and $$MSE=n^{-1}\sum_{i=1}^n (\hat{y}_i - y_i)^2$$ the mean squared error. Furthermore $$MAE=n^{-1}\sum_{i=1}^n |\hat{y}_i - y_i |$$ is called the mean absolute error.

Does anybody know the name and meaning of this measure of model fit?

$$n^{-1}\sum_{i=1}^n |\hat{y}_i| - | y_i |$$

It has the interesting property to measure how close the estimate is below or above the true value in absolute terms. This may be useful to measure e.g. the extent of regularization towards zero. I wonder if this measure is discussed anywhere in the statistical learning literature and has a name.

Example. Consider the plot below. Black illustrates the true functional form how X relates to Y. Red and green are predictions from two different models. Clearly red has bias, which is positive on the left and negative on the right. Green has less bias but larger variance than red. If I calculate B I find -0.025 for red and -0.022 for green. The MAE is 0.084 for red and 0.114 for green. The root of MSE is 0.098 for red, and 0.142 for green. The suggested measure gives -0.039 for red and 0.005 for green. The measure indicates average shrinkage to zero for red but not for green although MAE-red is lower than MAE-green. It works here because for most pairs of values there is no sign change; if there were a sign changes, the measure probably fails as noted by @Ruben van Bergen.

My question can thus be restated: how can I properly measure bias (or absolute bias, bias-squared) in this setting?

enter image description here

$\endgroup$
  • 4
    $\begingroup$ Doesn't the MAE already have that property? And more importantly, doesn't your suggested measure actually fail to work this way? Suppose, for instance, that the true value of y is 100, and the estimate is -100. Your measure would then return a value of 0, even though the error is actually very large. This seems undesirable for a measure of goodness-of-fit. $\endgroup$ – Ruben van Bergen Jun 4 at 9:06
  • $\begingroup$ @RubenvanBergen MAE incorporates also the variance. Yes the measure fails in your situation. I will give an example to make the problem clearer. $\endgroup$ – tomka Jun 4 at 10:22
  • $\begingroup$ @RubenvanBergen Example added. My question can also be re-stated as how to estimate the average absolute / squared bias in this setting. My measure probably is not ideal in general but works if there is no sign change for pairs of estimates and true values. $\endgroup$ – tomka Jun 4 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.