I know this question has been asked before (e.g. here Meaning of 'number of parameters' in AIC), but I am still confused. What exactly makes something as a parameter for the AIC penalty, specifically in the context of time series forecasting? The answer I linked says it relates to the number of "estimated parameters," but I'm not sure what qualifies as "estimating." Is calculating an average estimating? I guess let me ask with some examples (making forecasting model Y from history X, both length n, with error e having no mean and variance sigma):
- A simple average, e.g. Y = m + e, m = sum(X)/n. This seems like it has only 1 parameter (sigma), but I also calculate m from the inputs; is that 'estimating,' bringing the parameter count to 2?
- A moving average, tuning the parameter L for length of recent history to consider: Y_i = sum(X_(i-L):X_(i-1))/L + e. Like above, this seems like it has 2 parameters (sigma and L), and it doesn't make sense to consider the n calculated means as parameters too.
- Linear regression, Y_i = a * i + b + e, calculating with simple regression a = sum((i-i_bar) * (x-x_bar)) / sum((i-i_bar)^2), b = x_bar - a * i_bar. Here _bar indicates a mean, with i_bar obviously being equal to (n+1)/2. Since again I am calculating and not estimating, it seems like this also only has 1 parameter, the sigma in e.
- Linear regression Y_i = a * i + b + e like above, but instead of calculating a and b with regression, I estimate them using a Nelder-Mead simplex. Since these are now estimated parameters, it seems like this model has 3 parameters.
Obviously, methods 3 and 4 will get nearly identical values for a, b and sigma. If "estimated parameters," in the context of AIC, means not including 'calculated' parameters, we would have AIC(model4) = -2 * log(L) + 2 * 3 = n * log(sigma) + 6, and AIC(model3) = -2 * log(L) + 2 = n * log(sigma) + 2. As I said, these methods produce (nearly) identical models, so it seems odd to me that they would entail different amounts of information loss. Furthermore, I don't understand how calculating a and b using regression from the historical values X vs. calculating them using a simplex from the historical values X is functionally different. Or for a possibly more relevant example, in exponential smoothing I can calculate the initial condition of position, velocity or seasonality using regression or estimate them with a simplex, and I have seen different authors treat these differently with regards to AIC parameter count (e.g. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.216.9030&rep=rep1&type=pdf) On the other hand, if 'calculated parameters' are included in "estimated parameters," the implications for models 1 and 2 make less sense (and don't match what I have seen in other sources).