1
$\begingroup$

I am using the corrected AIC to select the lag order in a simple AR(p) model. I chose the the AICc since my sample is fairly small (n=135). The AICc minimal model is the AR(15). To me it seems like an overfit to include 15 lags in such a small sample.

Do you agree? And does anyone know a rule of thumb for max lag order relative to sample size?

From the arfunction in the stats package in R:

order.max
Maximum order (or order) of model to fit. Defaults to the smaller of N-1 and 10*log10(N) where N is the number of observations except for method="mle" where it is the minimum of this quantity and 12.

To me these default make little sense. Where do they come from?

$\endgroup$
5
  • $\begingroup$ I have heard that decreasing between 3 and 5 degrees of freedom can help, but I also have dealt with spline methods where I can have effective non-integer parameter values, and so in that case it gets messier. $\endgroup$ May 20 '13 at 21:58
  • 1
    $\begingroup$ Care to be more specific about the data? If so, what is the series being modeled? Have you tried inspecting the acf and pacf? You might find a more parsimonious model if you try doing that. $\endgroup$ May 20 '13 at 22:34
  • 1
    $\begingroup$ Also, by using the ar() function, are you deciding a priori that an AR model provides an adequate representation of the data generating mechanism? In other words, does the ar() function rule out the possibility of adding MA, seasonal AR, seasonal MA, and "difference" terms to the model? $\endgroup$ May 20 '13 at 22:43
  • $\begingroup$ I am using it to forecast monthly house prices. I have inspected the acf and pacf, but they dont give much information since the AR(p) does not seem to fit the data well. That is, yes I'm deciding apriori that the model is a good fit. The reason for this: I have a set of models (ARIMA and exponential smoothing models). The AR(p) is only included as a benchmark in the set, as is quite common in the forecasting literature. $\endgroup$
    – tfunk
    May 20 '13 at 22:48
  • $\begingroup$ AICc is substantially more conservative than the uncorrected AIC in small samples. If it's still choosing too large a model for your purposes, you might be better served by BIC, which is more conservative than AIC -- but on sufficiently small samples there's no guarantee it won't choose a larger model than AICc. $\endgroup$
    – Glen_b
    May 21 '13 at 1:23
1
$\begingroup$

You need to use the likelihood for the whole sample from the first principles based on $$ \log L \sim -\frac12 ({\bf y}-\mathbf{\mu})'\Sigma(\theta)^{-1}({\bf y}-\mathbf{\mu}) - \frac12 \log |\Sigma(\theta)| $$ where ${\bf y}\in \mathbf{R}^{135} $ is your whole sample vector, and $\Sigma(\theta)$ is the model-implied covariance matrix of your ARMA($p,q$) process. God only knows what your naive AIC calculation for i.i.d. data is actually doing; it is out of context and has little value here.

$\endgroup$
3
  • $\begingroup$ How do you know how he's calculating the AIC? $\endgroup$ May 21 '13 at 22:55
  • 1
    $\begingroup$ I am using the Arima function (forecast package) which is largly a wrapper for the arima function (stats package) to fit the model. From the description in R: "The exact likelihood is computed via a state-space representation of the ARIMA process, and the innovations and their variance found by a Kalman filter." I am not sure how this compares to your formula. $\endgroup$
    – tfunk
    May 21 '13 at 23:34
  • $\begingroup$ I don't know, either, although from your description, it could be inferred that the likelihood that your model returns is the right figure (computed differently, though, as the sum of squares of uncorrelated innovations, Kalman-filtered out). The reference you gave is for i.i.d. data, where the sum of squares is taken in place of the likelihood, that's why I decided to write my answer up. $\endgroup$
    – StasK
    May 23 '13 at 2:45
1
$\begingroup$

AIC's for selecting the "best" model from a candidate set*, so consideration of overfit has to be relative to the other models included in that set. I'd wager the AR(15) model has substantial overfit compared to other models not included in the candidate set, such as those suggested by @Graeme (an MA(1) model is equivalent to an AR($\infty$) model), but if for some reason you're not interested in those, there's no reason not to stick with it as the best of a bad lot.

*by estimating which has the least expected Kullback–Leibler divergence from the true model

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.