2
$\begingroup$

I'd like to produce forecasts by considering different scenarios. I'm using Mean Error (ME), where the error $=$ forecast $-$ demand, and Mean Square Error (MSE) to evaluate the results. For the scenarios that bias (ME) is negative the MSE is very high, how can I interpret these results?

I know that the MSE$=$variance of forecast error + bias$^2$, so for these scenarios, we have a low bias but MSE is high, so it means the variance of forecast error is high.

enter image description here

enter image description here

$\endgroup$
6
$\begingroup$

The units of MSE will be whatever the units of the Error are, squared; so metres squared if your forecast is in metres; (tons per hectare) squared if your forecast is in tons per hectare. Even if your units are counts, squared counts are not directly comparable.

The usual remedy for this is to work with Root MSE (RMSE) to get back to the original units.

Whether RMSE is a good metric for forecasting assessment is a different and delicate matter.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you, even if we use RMSE, for the scenarios that ME is negative , RMSE has the highest values, why? I'm trying to find a intuitive explanation $\endgroup$ – Roji Jun 27 '13 at 8:21
  • $\begingroup$ "... for the scenarios that ME is negative..." this makes me wonder if you using the mean of the error, or the mean of the absolute value of the error? Because if it's the former, positive and negative errors will cancel out. A set of errors {-100, 100, -100, 100} has a mean error of zero, despite every entry having an absolute error of 100. However the root mean squared error will be 100, as $(-100)^2 = 10000$, i.e. positive and negative values don't cancel. Whether you want errors to cancel will depend on your application. Most applications don't, so use mean squared or mean absolute error $\endgroup$ – Pat Jun 27 '13 at 8:59
2
$\begingroup$

As we do not know the method you use, it's difficult to explain your findings. However it looks for me as a typical bias-variance trade-off case. So there is a theory behind model selection using MSE: biased estimators can produce lower variance and vice-versa.

http://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.