So I have a predictive model generating a list of $\hat{y_i}$, and the error of each forecast is $\hat{y_i}-y_i$.

I would like to measure the variance of the errors. This can be calculated by $$\frac1{n-1}\sum_{i=1}^n\sqrt{(e_i-\bar{e})^2} $$ where $e_i = \hat{y_i}-y_i$.

The motivation to use this metric is I would like my model generating relatively stable errors. I explored a little bit but cannot find the correct terminology. Is there a statistical concept for this? Does this motivation make sense at all? Any advice is appreciated.

  • $\begingroup$ What if one of the errors is less than the mean of the n errors? Do you really want the square root of a negative difference? $\endgroup$ – Ed V Jul 22 '19 at 2:27
  • $\begingroup$ Ooops I forgot to type the square... fixed! $\endgroup$ – Rachel Zhang Jul 22 '19 at 3:28
  • $\begingroup$ I'm not quite clear on the motivation here. In what context would this be used? Could you provide a toy example? Also, have you considered examining homogeneity of variance? As written, the metric doesn't tell me much about the predictions. But, if you plotted the errors against the predicted values, then you could assess the homogeneity of variance (this sort of plot is used a lot in regression problems). $\endgroup$ – Demetri Pananos Jul 22 '19 at 4:14
  • $\begingroup$ This is a predictive model and besides bias and accuracy, I would also measure their variance - but not the variance of the predictions, I would like to see if their predicting errors are stable enough. It is a model used in the business industry so a relatively stable model would be more useful. @DemetriPananos $\endgroup$ – Rachel Zhang Jul 22 '19 at 11:49

I understand why you might want a "stable" model, but your concept still needs to be fleshed out a little more. To collapse the variance of the prediction errors onto a single number without taking into account any other information can lead to erroneous conclusions.

Let me explain.

Let's say you are using age as your only feature in your model to predict how much your customer will spend at some retail store. The variance of spend changes with age. Eighteen year old customers don't have a lot of money, so they all spend more or less the same amount. However, 40 year old customers vary wildly in their incomes, and so those with more money will spend more. Clearly, spend is heteroskedastic (i.e. has non-constant variance).

Now, assume that you train a model and deploy it to production. In the first month after deployment, lots of 18 year olds come and buy in the store. Since variance of spends for 18 year olds is small, your model's variance in predictions is small.

But then suppose in the second month, a lot fo 40 year olds come and spend. By virtue of their spend being highly variable, your model's errors will also be highly variable! It will look as if your model has degraded (or, in your own words, become unstable) when in reality nothing has changed.

All this to say, if you are looking for measure the variance of your prediction errors, you need to condition on something. Usually, you look at the error as a function of the predictions (like I've mentioned in my comment).

So, if by "stable errors", you mean that the variance of errors is constant across predictions (i.e. the errors are homoskedastic), then makes more sense. If that is not what you mean, I think the idea needs to be fleshed out a little more for me.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.