AIC calculation with very low negative log likelihood

Question

I am using AIC formula (AIC=2k−2lnL) to compare different exponential models.

I know that this formula is used to penalize complexed models (with high number of parameters).

The problem I have is that the negative log likelihood term (-2lnL) is very low (order of -1.5e50). Thus, it does not have the same scale as the first term of the AIC (2k, k=5 or k=8) and in consequence does have a larger impact on calculating AIC.

As a consequence, AIC cannot in my case select the best performing model based on both the number of parameters and the negative log likelihood.

Is it normal to have this usecase? Is there an alternative to AIC for my usecase?

Answer 1

As a consequence, AIC cannot in my case select the best performing model based on both the number of parameters and the negative log likelihood.

I do not see why this should be the case. If the likelihood is meaningful and is calculated correctly, AIC will be meaningful and will serve its role, i.e. estimate $-2n \ \times$ the log-likelihood of a new data point from the same data generating process / population. You can still use AIC for model comparison. It is not uncommon for the likelihood term to dominate the penalty term. This is natural when the estimation precision is sufficiently high (e.g. due to lots of data or strong signal relative to noise) so that there is little overfitting and a penalty that is small relative to the log-likelihood is sufficient to account for it.

Answer 2

The log likelihood of your data is the sum of the log likelihood of each individual data point, all of which will be $\lt 0$. This means that unless your model is a very bad fit to the data, an extremely low log likelihood reflects the fact that you have an enormous number of data points.

Now, AIC is supposed to approximate out of sample predictive accuracy: a model with lower AIC should make better predictions based on new data than a model with higher AIC, given particular assumptions. It does this by finding a balance between overfitting (just picking the model that best fits the training data - that has the lowest log likelihood) and underfitting (picking the model with fewer parameters).

$$\text{AIC} = 2 k - 2 \text{ln}(\hat L)$$

Fortunately, the more data you have, the less you need to worry about overfitting. A model with lots of parameters will overfit on a small training dataset, but work fine on a larger dataset. This is why as the size of the dataset grows, and the magnitude of the log likelihood term increases, AIC depends more on how well the model fits the training data (log likelihood), and less on the number of parameters.

On the other hand, there's the BIC, which is supposed to approximate the Bayes Factor.

$$\text{BIC} = \text{ln}(n) k - 2 \text{ln}(\hat L)$$

BIC is supposed to find which model is actually true, not which model makes the most accurate predictions. To avoid just being driven by the log likelihood in cases where there is a huge amount of data, the penalty applied on the number of parameters, $k$, increases as a function of the log of the number of data points, $n$.