1
$\begingroup$

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following:

  • I have two input variables (A and B), and a single target variable. All variables are time series.
  • Target variable is modeled with GLM (Poisson)
  • AIC is used to compare models built with the following input variables:
    • Just A
    • A and B
    • A and B, both averaged over some window, with some lag (i.e. 4 hyperparameters, taken into account when calculating the AIC value). Here, AIC also determines the best hyperparameters (with grid search)

I would like to ask for comments on the following:

  1. Usually, I would perform cross-validation in order to determine the best hyperparameters and to choose the best model. CPU time savings with AIC makes the above (AIC based procedure) almost too good to be true... So - is the above (AIC based) approach correct? Given AIC and cross-validation equivalence for linear models, I would say yes.

  2. If "A and B" based model (raw or windowed) gets chosen with AIC criterion, I want to estimate the contribution towards the target variable coming from B (as sum of B values times the corresponding coefficient). My intuition is that such approach is wrong, slightly resembling the problem with the in-sample error estimation. Should I implement double cross-validation like setup, with the inner cross-validation replaced by the above (AIC based) procedure?

$\endgroup$
1

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.