2
$\begingroup$

I am using AIC formula (AIC=2k−2lnL) to compare different exponential models.

I know that this formula is used to penalize complexed models (with high number of parameters).

The problem I have is that the negative log likelihood term (-2lnL) is very low (order of -1.5e50). Thus, it does not have the same scale as the first term of the AIC (2k, k=5 or k=8) and in consequence does have a larger impact on calculating AIC.

As a consequence, AIC cannot in my case select the best performing model based on both the number of parameters and the negative log likelihood.

Is it normal to have this usecase? Is there an alternative to AIC for my usecase?

$\endgroup$
2
  • $\begingroup$ Do you have an enormous number of data points? $\endgroup$
    – Eoin
    Nov 25, 2020 at 21:35
  • $\begingroup$ AIC is too simple a measure for model selection. See the discussion on AIC for multiple linear regression, and an alternative to it, $SIC_f$, on: stats.stackexchange.com/questions/524258/… $\endgroup$ Jun 6, 2021 at 10:46

2 Answers 2

2
$\begingroup$

As a consequence, AIC cannot in my case select the best performing model based on both the number of parameters and the negative log likelihood.

I do not see why this should be the case. If the likelihood is meaningful and is calculated correctly, AIC will be meaningful and will serve its role, i.e. estimate twice the negative log-likelihood of a new data point from the same data generating process / population. You can still use AIC for model comparison. It is not uncommon for the likelihood term to dominate the penalty term. This is natural when the estimation precision is sufficiently high (e.g. due to lots of data or strong signal relative to noise) so that there is little overfitting and a penalty that is small relative to the log-likelihood is sufficient to account for it.

$\endgroup$
1
$\begingroup$

The log likelihood of your data is the sum of the log likelihood of each individual data point, all of which will be $\lt 0$. This means that unless your model is a very bad fit to the data, an extremely low log likelihood reflects the fact that you have an enormous number of data points.

Now, AIC is supposed to approximate out of sample predictive accuracy: a model with lower AIC should make better predictions based on new data than a model with higher AIC, given particular assumptions. It does this by finding a balance between overfitting (just picking the model that best fits the training data - that has the lowest log likelihood) and underfitting (picking the model with fewer parameters).

$$\text{AIC} = 2 k - 2 \text{ln}(\hat L)$$

Fortunately, the more data you have, the less you need to worry about overfitting. A model with lots of parameters will overfit on a small training dataset, but work fine on a larger dataset. This is why as the size of the dataset grows, and the magnitude of the log likelihood term increases, AIC depends more on how well the model fits the training data (log likelihood), and less on the number of parameters.

On the other hand, there's the BIC, which is supposed to approximate the Bayes Factor.

$$\text{BIC} = \text{ln}(n) k - 2 \text{ln}(\hat L)$$

BIC is supposed to find which model is actually true, not which model makes the most accurate predictions. To avoid just being driven by the log likelihood in cases where there is a huge amount of data, the penalty applied on the number of parameters, $k$, increases as a function of the log of the number of data points, $n$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.