I trained two basic feed-forward neural networks on time series data. The first one uses the observation at time step $t$ to predict $t+1$. Hence, it only has one predictor variable. The second network uses a temporal lag of size 1, i.e. it uses the observation at time step $t$ and $t-1$ to predict $t+1$. Hence, it uses two predictor variables.

Comparing the MSE of both models reveals, as expected, that the second network (the one that with the temporal lag) predicts better. However, the first model yields the lower AIC, probably because it has less parameters (I calculated the likelihood function of the models using the number of samples and the MSE).

If I compare the 10-fold cross-validated (CV) MSE of the models instead of their AICs, the second model is preferred, despite its larger number of parameters.

So, which model should I choose? AIC says the first model, CV MSE the second one.

  • 1
    $\begingroup$ Interesting... Asymptotically AIC and MSE should select the same model, if I am not mistaken. Perhaps there is some problem with likelihood in AIC, e.g. the assumed error distribution is far from the realized residual distribution? If you used MSE for likelihood calculation, I suppose you are assuming normal errors. $\endgroup$ – Richard Hardy Jan 15 '16 at 19:07
  • 1
    $\begingroup$ I was imprecise: AIC equals leave-one-out CV, not K-fold CV, asymptotically. It is BIC that matches K-fold CV asymptotically. AIC should be relevant for forecasting, while BIC is better suited for recovering a true model. $\endgroup$ – Richard Hardy Mar 13 '17 at 19:42
  • $\begingroup$ Thanks. In my opinion LOO CV is pretty bad in practice because of the extremly high variance of the resulting models. So, the same should also apply to AIC, right? In my experience, AIC is more often used than BIC, but 10-fold CV is more often used than LOO CV. Why is BIC so rarely used though it will results in lower variance than AIC? $\endgroup$ – Julian Mar 14 '17 at 8:10
  • $\begingroup$ I don't know why LOOCV does not work for you, but AIC is asymptotically optimal for one-step-ahead forecasting under square loss (while BIC is not), and LOOCV is supposed to be asymptotically equivalent to AIC. That is all I know... $\endgroup$ – Richard Hardy Mar 14 '17 at 8:24
  • $\begingroup$ Thanks Richard. Sorry, I was commenting on the general case, not time series. For time series, AIC might the way to go but I am unsure about the general regression case... but I think this goes beyond my original question. $\endgroup$ – Julian Mar 14 '17 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.