3
$\begingroup$

Having two models to describe the data, one computes for each of them $$AIC=2k-2L,$$ where $L$ is the max log-likelihood and $k$ the number of parameters in the model. Then we choose the model with smaller $AIC$ and evaluate its strength against the other model via $\Delta AIC$.

Is it really correct to use the same formulation of $AIC$ for uni- and multivariate distributions? E.g., in a 1D setting one can compare a Gaussian and a mixture of 2 Gaussians, and in 2D the models can be a bivariate Gaussian and a mixture of two bivariate Gaussians. Why should the above formula for $AIC$ be the same regardless of the dimensionality of the data? Shouldn't there be some way (e.g., an additional term) dependent on the number of dimensions?

$\endgroup$

2 Answers 2

2
$\begingroup$

Why should the above formula for AIC be the same regardless of the dimensionality of the data? Shouldn't there be some way (e.g., an additional term) dependent on the number of dimensions [of the data]?

There is no need for an additional term, since the maximised value of the likelihood function (which appears in the AIC formula) already maximises over all possible likelihood values in the model, corresponding to all possible sampling distributions, even if these posit different dimensions for the data. It is crucial to remember that the data vector is observed, which means that the length of the data vector (i.e., the data dimension) is also observed and therefore known.

The sampling density of the observed data vector must be zero under any model that does not yield the observed data dimension. Therefore, if the model contains sampling densities giving different dimensions for the data, the likelihood function will be zero over any set of parameter values representing sampling distributions that would give a data dimension different to the observed dimension. The result is that any model form that posits different possible dimensions for the data, will immediately collapse to the correct (observed) dimension, once you maximise the likelihood function over the entire parameter space.


An illustrative example: For most models of interest, the dimension of the data is fixed by the length of the observed vector in the likelihood function. However, suppose we generalise from this and allow a model form that does not fix the data dimension. To see what happens in an example, consider a generalised form of an IID Gaussian model $\{ \mathcal{M}_{n, \mu, \sigma} | n \in \mathbb{N}, \mu \in \mathbb{R}, \sigma \in \mathbb{R}_+ \}$ where the length of the observed data vector is able to differ within the model, and is described by the parameter $n$. For any observed data vector $\mathbf{x} = (x_1,...,x_k)$ (which has dimension $k$), the sampling density for each possible parameterisation in this model is:

$$p(\mathbf{x} | n, \mu, \sigma) = \mathbb{I}(k=n) \times \prod_{i=1}^n \text{N}(x_i | \mu, \sigma^2).$$

For this particular model, the corresponding likelihood function can be simplified down to:

$$L_\mathbf{x}(n, \mu, \sigma) = \begin{cases} (2 \pi)^{-n/2} \sigma^{-n} \exp \Bigg( - \frac{1}{2 \sigma^2} \sum_{i=1}^n (x_i-\mu)^2 \Bigg) & & \text{if } k = n, \\[6pt] 0 & & \text{if } k \neq n. \\[6pt] \end{cases}$$

Now, we can clearly see that $L_\mathbf{x}(n, \mu, \sigma) = 0$ for all $n \neq k$. Hence, if we maximise over the entire parameter space, we get:

$$\hat{L} \equiv \max_{n, \mu, \sigma} L_\mathbf{x}(n, \mu, \sigma) = \max_{\mu, \sigma} L_\mathbf{x}(k, \mu, \sigma).$$

That is, the maximised likelihood value reduces down to the maximised value over the parameter subspace where $n=k$ (i.e., where the data dimension in the model corresponds to the observed dimension of the data vector.). The corresponding AIC is:

$$\text{AIC} = 2k - \hat{L} = 6 - \max_{\mu, \sigma} L_\mathbf{x}(k, \mu, \sigma).$$

We can see here that the subspace of the model with $n \neq k$ is effectively irrelevant, since it has zero likelihood. The only effect of including the additional parameter $n$ in the model is to add one additional useless parameter, which increases the AIC by two points (since it adds one parameter to the model, without raising the maximum likelihood value). This explains why we usually do not bother specifying the model with this wide a scope. By restricting this generalised model to a standard model where we restrict our scope to match the observed dimension of the data, we remove the useless parameter describing dimension, and get a better AIC.

As you can see, even when we consider a broad model form that allows for different possible dimensions of the data, there is still not a problem with the AIC (and indeed, it is helpful, because it tells us not to include this additional useless space in the model that contradicts the observed data). Because the sample data is observed we observe the dimension of the data. Any distribution that would give a data vector with a different dimension than the one that was observed has zero likelihood --- i.e., it is contradicted by the observed data. Any useful measure for fit for model comparison will therefore count out this model as the worst possible model. The AIC meets this criterion --- if the observed dimension of the data contradicts the model then $L=0$ over the parameter space for that model, and so $\text{AIC} = \infty$, which is the worst possible value.


As a final point, it is worth noting that even if one has a philosophical objection to pre-specifying the data dimension in the model (and there are certainly reasonable arguments for this view), this still would not detract from the value of the AIC as a model comparison tool. Suppose that you were to decide that it is invalid post hoc cherry-picking to posit a model that fixes the data dimension to be equal to the observed dimension, and that every valid model must therefore instead include the free parameter $n \in \mathbb{N}$ to describe all possible values of the data dimension (prior to determination of the sample size). Even in this case, all that would happen is that every model now includes one additional parameter, without changing the maximised likelihood value. So now every model has its AIC shifted upward by two points, but the difference in AIC between models remains the same. This means that the AIC remains just as useful as a comparative tool between different models.

$\endgroup$
1
$\begingroup$

There are many corrected versions of AIC which aim at reducing bias. They typically depend on the dimension, the sample size, or the number of covariates in linear regression models. If you want to find a corrected AIC version in your context, you may need to specify more details on your model.

The use of AIC is not recommended to select the number of components in mixture models. See for instance:

Choice of the Number of Component Clusters in Mixture Models by Information Criteria.

Generating Gaussian Mixture Models by Model Selection For Speech Recognition

$\endgroup$
4
  • $\begingroup$ Thanks for the links, but that does not really answer my question, neither explaining why in the usual formulation there is no dimension-dependent term, nor how to take the dimensionality into account if required. $\endgroup$
    – corey979
    Commented Aug 15, 2016 at 14:50
  • $\begingroup$ @corey979 You are missing the point of AIC and "model selection" in general. Please think that some models may have infinite dimensionality, right? So are you saying that those models should be treated differently? Why is the dimension so important to you? How is it different from the number of predictors, number of volatility states, number of parameters explaining tails of the residual? It is all important. Each model is a suggestion, an expert opinion. Does not matter how that opinion has been formed. $\endgroup$
    – stans
    Commented Nov 5, 2019 at 6:19
  • $\begingroup$ @corey979 You compare the expert opinions based on their "relevance to the problem". You penalize self-indulgence, you encourage respect for the data. That is what modeling is all about. dim(Y) is just one parameter. $\endgroup$
    – stans
    Commented Nov 5, 2019 at 6:22
  • $\begingroup$ @corey979 Understood, and thanks. Done. $\endgroup$
    – stans
    Commented Nov 5, 2019 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.