Normalising likelihood for BIC/AIC calculation

Question

I am running some model inference using AIC and BIC. My problem is that when I go and calculate the (maximum) loglikelihoods of my models, they are usually really high (range between 4700 and 1400 approx.). This is mostly because I have a lot of data points, in the order of 20k, and (apparently) decently good models. So the likelihood for each individual point is often > 1 and the sum of their logs is positive and gets quite high.

Now, if I use these values to calculate BIC and AIC and from there the posterior probabilities of my model, I often get a numerical error, because I have very negative BIC/AIC ($<-1000$) and the number I'd get with $\exp(-0.5\text{BIC})$ is just too big for R. This is also the case when I use $\text{BIC}-\text{BIC}_{max}$.

I am considering using normalised likelihood instead of likelihood (i.e. $1/N\cdot \text{likelihood}$). How does this (and sample size in general) affect BIC/AIC and model inference methods? I couldn't really find anything useful to read about this.

Answer 1

I would advise against normalizing the likelihood by the number of observations, since this would make the definitions of the BIC and the AIC irrelevant. AIC/BIC are not arbitrary combinations of a likelihood term and a penalty term: the BIC is an approximation of the model evidence, while the AIC is an unbiased estimator of the Kullback-Leibler divergence between a model and the ground-truth. For instance, using the normalised likelihood would make the approximation $BIC_{\mathcal{M}} \approx -2\log p(\mathcal{D}|\mathcal{M})$ wrong.

Instead of manipulating "wrong" posteriors, you could manipulate "true" BIC scores. Most papers comparing models simply use the values of the BIC without transforming it into a posterior, see for instance Fig. 5 in Pfister, J. P., & Ghosh, A. A priority-based model for smartphone screen touches.

Since you have enough observations to split them between a training and a validation sets, another alternative to compare your models would be to use cross-validation instead of AIC/BIC.

Answer 2

Lower BIC scores are better, so the normalised value you're supposed to use is $\Delta BIC = BIC - BIC_{\text{min}}$, not max (see this paper, which shows the calculations for AIC, but the BIC equations are the same and this is easier to find).

With that done, your highest BIC will be $0$, everything else will be positive, and the $w_{BIC} = \exp(-.5\Delta BIC)$ term will work as expected.