1
$\begingroup$

I have a database of forecasts (ranging from 1 week to 4 weeks in advance) and one of experimentally recorded values for a specific index whose value ranges from 1 to 9 (most of the time it measures 2 or 3, with values 5 and upwards becoming exponentially more rare). I guess I should compare each delta-time separately (1-week, 2-weeks, etc.).

I'd like to get an idea on how accurate these forecasts tend to be. Would average standard deviation be the best way? Or would you suggest something else?

Thank you

Edit: given two sets of variables, A being forecasted number and B number measured, I was thinking about assessing forecast accuracy by doing something like:

$$ A_1 - B_1 = |C_1|$$ $$ A_n - B_n = |C_n|$$ $$ \frac{\sum_i |C_i| }{n} = D$$

And then...I don't know. Advice welcome. Thank you

$\endgroup$
5
  • $\begingroup$ Average standard deviation of what quantities? $\endgroup$
    – whuber
    Jan 30, 2020 at 20:19
  • $\begingroup$ Average SD of a single digit index variable. $\endgroup$
    – Domi
    Jan 30, 2020 at 20:27
  • $\begingroup$ When the forecast is a constant, its SD will be as small as possible--but obviously has little to do with accuracy. $\endgroup$
    – whuber
    Jan 30, 2020 at 20:32
  • $\begingroup$ I see, thank you. I am trying to understand how can I assess accuracy of forecasts by comparing the values of that index number in forecasts and the measured values after the even occurred. $\endgroup$
    – Domi
    Jan 30, 2020 at 20:34
  • $\begingroup$ What do you think about my answer? Does it answer your question? If so, you may accept it by clicking on the tick mark to the left. Otherwise, you may ask for further clarification. This is how Cross Validated works. $\endgroup$ May 2, 2020 at 6:34

1 Answer 1

2
$\begingroup$

Standard deviation is a poor measure of accuracy as it does not reflect bias. You can add bias to the forecasts, but this will not change the standard deviation. The same problem is characteristic to variance. Instead of variance, we use mean squared error. Instead of standard deviation it is then natural to use root mean squared error.

Responding to an edit: the quantity you have now defined is a well known measure of forecast accuracy, it is mean absolute error (MAE). If the loss/disutility of making a forecast error is linear and symmetric, it makes sense to use MAE.

$\endgroup$
2
  • $\begingroup$ The SD of the differences between the forecast and actual values, however, will reflect bias--and comments by the OP suggest this is what they have in mind. $\endgroup$
    – whuber
    Jan 30, 2020 at 21:01
  • $\begingroup$ @whuber, differences between forecasts and actual values are forecast errors, right? SD of forecast errors will not reflect the mean of the forecast errors, and it is this mean that corresponds to the bias of the forecasts. So I think my point remains valid. $\endgroup$ Jan 31, 2020 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.