I have forecasts for different sized samples using a variety of methods like DES (Double Exponential Smoothing), SES, MA and WA (Weighted Average) to test the strength of the forecasting models. The problem is that SES was used on 100 items, DES was used on 130 items, and so on. I have the forecasts for each item for each month in the year 2017, as well as the actual values for the same months. Since ideally the error should be as close to zero as possible for all cases, if the scaled Mean Error for SES is less than DES, is it valid to say that the SES should be ranked higher for forecasting ability than DES?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I reckon comparing estimators on samples of different size was the basic idea of Student's test, followed by the work of Fisher on the degree of freedom.
So the answer is YES, provided each sample is iid (identical independant distribution) and that you unbias the standard deviation by the correct degree of freedom. Unless you use extra parameters (seasonality, exogenous variable,...) compare
- $\sigma_{SES} = \sqrt{\frac {SSE} {n-1}}$ for the Single Exponential Smoothing (which estimates 1 parameter),
- $\sigma_{DES} = \sqrt{\frac {SSE} {n-2}}$ for the Double Exponential Smoothing (which estimates 2 parameters),
- $\sigma_{MA} = \sqrt{\frac {SSE} {n-1}}$ for the weighted or not weighted average (which estimates 1 parameters),
In the above $SS$ is the Sum of Square of the Error and $n$ is the sample size which of course dependent of the sample.
