3
$\begingroup$

When performing AIC (or BIC) model selection between competing models, one performs individual fits for all models (on the same training data set) and then compares their AIC (or BIC etc) values. If the difference between the lowest AIC value for some model is greater than 10 we can say that the model selection is conclusive. There is a great deal of discussion on the topic in Burnham and Anderson.

My problem: I am fitting two parametric models of different functional form (same number of parameters, same type of response), say $y = \rho_1(r|\theta_1)$, and $y = \rho_2(r|\theta_2)$ that give fits of similar quality and their AICc difference is on the order of $\Delta \mathrm{AICc}\sim 2$ (which is not conclusive). Now, I know that one of these models is the "true" one (in the sense that it was the model from which the data were created), and this comes out to have the best AICc value. Since both models have same number of parameters, it is essentially their mean square error loss that makes the selection. However if I use a sample of test data (after training), the difference of their (average) square error loss climbs up to $\sim 10$.

My questions:

a. Is it possible to use AICc with test data (i.e. evaluate square error loss on test data, but the parameters of the model were obtained from a training data set) and perform model selection? My intuition says yes, but I have no theoretical justification for this. In addition I have no reference cut off value for conclusive model selection if I use test data set.

b. Is there some kind of error based model selection on test data that can be conclusive (in the sense that AICc is conclusive when the difference $\Delta \mathrm{AICc} > 10$)?

PS. I understand that "all models are wrong", in the sense that they are approximations of the data realization and perhaps the very meaning of conclusive model selection doesn't make much sense.

|cite|improve this question
$\endgroup$
5
$\begingroup$

The quantity that AIC / AICc estimates is the expected out-of-sample log-likelihood (see Burnham & Anderson 2004, Sec. 2.2), $$ \mathbf E_y \mathbf E_x [ \log g(x | \hat \theta (y)) ], $$ (multiplied with $-2$). This formula means, you obtain maximum likelihood parameter estimates from one sample, $\hat \theta (y)$, then compute the logarithm of the likelihood $g$ of these parameters on another independent sample $x$ (from the same source), and then average across infinitely many realizations of both samples, $\mathbf E_y \mathbf E_x$.

Akaike's main result is that $\log g(x | \hat \theta (x)) - K$ is an asymptotically unbiased estimator of the quantity given above, where $K$ is the number of parameters. Equivalently, $$ \mathrm{AIC} = - 2 \log g(x | \hat \theta (x)) + 2 K $$ is an asymptotically unbiased estimator of $$ - 2 ~\mathbf E_y \mathbf E_x [ \log g(x | \hat \theta (y)) ]. $$ If the sample size is small compared to the number of parameters $K$, a better correction term (AICc; see Burnham & Anderson 2004, Sec. 7.7.6, and McQuarrie & Tsai 1998) may be necessary which is more complicated than $2K$, and it is different for different models (likelihood functions $g$).

If you use cross-validation with $$ - 2 \log g(x_\mathrm{test} | \hat \theta (x_\mathrm{train})) $$ as the error measure, you are estimating (almost) the same thing.

The differences between AIC / AICc and cross-validation are:
– AIC / AICc formulas are based on an approximation for large samples (they are only asymptotically correct).
– For cross-validation, in order to have test data $x_\mathrm{test}$ the training (estimation) data $x_\mathrm{train}$ must be smaller than the original sample $x$. Moreover, the cross-validated measure depends on two random samples instead of just one, so it may be noisier.

Answer a: AIC / AICc and cross-validation are alternative methods to do the same thing (if log-likelihood is the error measure). It therefore does not make sense to use them together.

Answer b: Since AIC / AICc and cross-validation with -2 log likelihood estimate (almost) the same quantity, it should be OK to apply the same scale to evaluate differences between models.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you very much for your answer, I am a bit confused. According to the definition from Burnham&Anderson 2002 (book, p. 61), for the AIC formula, $AIC = -2 Log(\mathcal{L}(\tilde{\theta}|y) + 2 K$, the quantity $\log \mathcal{L}(\tilde{\theta}|y)$ is the "numerical value of the log-likelihood at it's maximum point". In this expression, the estimated parameters $\tilde{\theta}$ are conditioned on the training data $y$. Then how can one use the AIC/AICc definition with unseen test data, i.e. different $y$ as you claim in your introductory paragraph? $\endgroup$ – Foivos May 21 '18 at 4:34
  • $\begingroup$ That's the definition of the criterion, but it is not the quantity estimated by AIC. I only have the second edition of the book (2004), there it is on the same page 61, in a box "The Key Result". As you can see, in this formula current data $x$ and data used to estimate the parameters $y$ are different. The difference between this expectation and -2 log likelihood is approximately $2K$. $\endgroup$ – A. Donda May 21 '18 at 20:15
  • $\begingroup$ @Foivos, I extended my answer to make the point of the previous comment clearer. $\endgroup$ – A. Donda May 21 '18 at 20:31
  • $\begingroup$ Thank you very much for your answer. So, to summarize my understanding of your answer: a) Since AIC is an approximation to the KL divergence, we can compare two models trained on the same fixed data set (section 2.11.1, box p. 81 of the book) but estimate AIC on different/independent data (x,y). b) Due to the inherent variance in the (independent samples of) data, we also expect some variance in the AIC estimated values. I have to note that in the majority of the examples I've seen people use AICc with training data and compare models (e.g. Elements of stat. learn p.231.) $\endgroup$ – Foivos May 22 '18 at 2:27
  • $\begingroup$ From the Elements of Statistical Learning (2nd edition, 12th printing, p. 231, Eq. 7.30): $ AIC(a) = \overline{err}(a) + 2\frac{d(a)}{N} \tilde{\sigma}^2_{\varepsilon}$, $\overline{err}(a)$ is the training average error. $\endgroup$ – Foivos May 22 '18 at 2:30
1
$\begingroup$

a. Is it possible to use AICc with test data

Information criteria shouldn't be necessary if you are evaluating models on a hold-out dataset. You can just choose the model that produces the lowest error (mean squared error, mean absolute error, or whatever metric you prefer) on the test data.

b. Is there some kind of error based model selection on test data that can be conclusive

'Conclusive' is unfortunately a bit fuzzy. Any model selection procedure (and any statistical test) retains some probability of being incorrect, or less accurate than the alternatives. But to basically repeat my answer to the previous question: the most robust approach is generally to evaluate your models' prediction errors through cross-validation. This question has a list of cross-validation techniques that can get you started: Compendium of cross-validation techniques

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.