I know this question has been asked before (e.g. here Meaning of 'number of parameters' in AIC), but I am still confused. What exactly makes something as a parameter for the AIC penalty, specifically in the context of time series forecasting? The answer I linked says it relates to the number of "estimated parameters," but I'm not sure what qualifies as "estimating." Is calculating an average estimating? I guess let me ask with some examples (making forecasting model Y from history X, both length n, with error e having no mean and variance sigma):

  1. A simple average, e.g. Y = m + e, m = sum(X)/n. This seems like it has only 1 parameter (sigma), but I also calculate m from the inputs; is that 'estimating,' bringing the parameter count to 2?
  2. A moving average, tuning the parameter L for length of recent history to consider: Y_i = sum(X_(i-L):X_(i-1))/L + e. Like above, this seems like it has 2 parameters (sigma and L), and it doesn't make sense to consider the n calculated means as parameters too.
  3. Linear regression, Y_i = a * i + b + e, calculating with simple regression a = sum((i-i_bar) * (x-x_bar)) / sum((i-i_bar)^2), b = x_bar - a * i_bar. Here _bar indicates a mean, with i_bar obviously being equal to (n+1)/2. Since again I am calculating and not estimating, it seems like this also only has 1 parameter, the sigma in e.
  4. Linear regression Y_i = a * i + b + e like above, but instead of calculating a and b with regression, I estimate them using a Nelder-Mead simplex. Since these are now estimated parameters, it seems like this model has 3 parameters.

Obviously, methods 3 and 4 will get nearly identical values for a, b and sigma. If "estimated parameters," in the context of AIC, means not including 'calculated' parameters, we would have AIC(model4) = -2 * log(L) + 2 * 3 = n * log(sigma) + 6, and AIC(model3) = -2 * log(L) + 2 = n * log(sigma) + 2. As I said, these methods produce (nearly) identical models, so it seems odd to me that they would entail different amounts of information loss. Furthermore, I don't understand how calculating a and b using regression from the historical values X vs. calculating them using a simplex from the historical values X is functionally different. Or for a possibly more relevant example, in exponential smoothing I can calculate the initial condition of position, velocity or seasonality using regression or estimate them with a simplex, and I have seen different authors treat these differently with regards to AIC parameter count (e.g. http://citeseerx.ist.psu.edu/viewdoc/download?doi= On the other hand, if 'calculated parameters' are included in "estimated parameters," the implications for models 1 and 2 make less sense (and don't match what I have seen in other sources).



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.