3
$\begingroup$

I'd like to produce forecasts by considering different scenarios. I'm using Mean Error (ME), where the error $=$ forecast $-$ demand, and Mean Square Error (MSE) to evaluate the results. For the scenarios that bias (ME) is negative the MSE is very high, how can I interpret these results?

I know that the MSE$=$variance of forecast error + bias$^2$, so for these scenarios, we have a low bias but MSE is high, so it means the variance of forecast error is high.

enter image description here

enter image description here

$\endgroup$

2 Answers 2

6
$\begingroup$

The units of MSE will be whatever the units of the Error are, squared; so metres squared if your forecast is in metres; (tons per hectare) squared if your forecast is in tons per hectare. Even if your units are counts, squared counts are not directly comparable.

The usual remedy for this is to work with Root MSE (RMSE) to get back to the original units.

Whether RMSE is a good metric for forecasting assessment is a different and delicate matter.

$\endgroup$
2
  • $\begingroup$ Thank you, even if we use RMSE, for the scenarios that ME is negative , RMSE has the highest values, why? I'm trying to find a intuitive explanation $\endgroup$
    – Roji
    Jun 27, 2013 at 8:21
  • $\begingroup$ "... for the scenarios that ME is negative..." this makes me wonder if you using the mean of the error, or the mean of the absolute value of the error? Because if it's the former, positive and negative errors will cancel out. A set of errors {-100, 100, -100, 100} has a mean error of zero, despite every entry having an absolute error of 100. However the root mean squared error will be 100, as $(-100)^2 = 10000$, i.e. positive and negative values don't cancel. Whether you want errors to cancel will depend on your application. Most applications don't, so use mean squared or mean absolute error $\endgroup$
    – Pat
    Jun 27, 2013 at 8:59
2
$\begingroup$

As we do not know the method you use, it's difficult to explain your findings. However it looks for me as a typical bias-variance trade-off case. So there is a theory behind model selection using MSE: biased estimators can produce lower variance and vice-versa.

http://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.