# Compare GLM AICs with different likelihoods?

If I have a generalized linear model (GLM) with a particular likelihood, and I have another GLM of the same data (say nested within the first model), I can compare the model performance using Akaike information criterion (AIC).

Let's say that I fit a Poisson GLM.

However, I wonder if I might get a better fit by using a negative binomial distribution.

Assuming the same features, what, if any, meaning is there to the difference in AIC values between the Poisson model and the negative binomial model? That is:

$$\log(\mathbb E[Y\vert X]) = \alpha_0 +\alpha_1X \implies AIC_1\text{, based on Poisson likelihood}\\ \log(\mathbb E[Y\vert X]) = \beta_0 +\beta_1X \implies AIC_2\text{, based on negative binomial likelihood}$$

Is there any meaning to comparing $$AIC_1$$ and $$AIC_2?$$ The answer here seems to indicate that it is not meaningful to compare $$AIC_1$$ and $$AIC_2$$, but the comments express some dissent. Kjetil's answer here seems to indicate that such a comparison would not be meaningful for a Poisson vs a Gaussian (for example) likelihood, due to the discrete vs continuous dominating measures, but for Poisson vs negative binomial, both dominating measures are discrete.

(A similar setup would be if it is meaningful to compare a linear regression fitted by minimizing square loss (maximum likelihood estimation for a Gaussian likelihood) and a linear regression fitted by minimizing absolute loss (maximum likelihood estimation for a Laplace likelihood). That seems like comparing MSE and MAE (or RMSE and MAE), which seems like an unfair comparison. At the same time, both dominating measures would be continuous.)

The Aikake information criterion (AIC) is derived by minimizing the Kullback–Leibler (KL) divergence between the data-generating distribution $$g(y)$$ and an approximating model $$f_\theta(y)$$ with parameters $$\theta$$. Asymptotically, KL$$(g \| f_\theta)$$ is minimized by the maximum likelihood estimator $$\hat{\theta}_n$$.

So we can use AIC to compare two generalized linear models (GLMs) — even if the likelihood functions are different — if we use (a) maximum likelihood and (b) the same data to estimate the model parameters $$\theta_1$$ and $$\theta_2$$. The comparison will tell us which model, $$f_{\theta_1}$$ or $$f_{\theta_2}$$, is the "closer approximation" for the truth $$g$$.

The theory behind AIC doesn't make assumptions about the form of the models $$f_{\theta_1}$$ and $$f_{\theta_2}$$, except the usual "mild regularity conditions". "Same data" means the same observations (data points) and the same outcome variable for each observation. Basically, the same data-generating distribution $$g(y)$$. That's why we can't compare a model for $$y$$ and one for a transformation such as $$\log(y)$$; a model for a continuous $$y$$ and one for a discretized version of $$y$$; two models with different supports for $$y$$, etc.

• Great answer! Thus, we can also compare different link functions like this, right? Commented Jul 30, 2022 at 10:30
• @MichaelM Yes, I think so: A different link function "only" changes the mapping between the predictors and the mean of the response, ie, it changes the model $f_\theta$. That being said, there are often better ways to choose the link function than the AIC. It took me a bit of time but I remembered a good example: take a look at Section 11.2 in the Regression Modeling Strategies course notes. Commented Jul 30, 2022 at 10:51
• Definitely! Still, I recently was asked exactly this one and was unsure. Commented Jul 31, 2022 at 7:11
• I’ll be posting a follow-up to clarify some remaining confusion I have, but this answers the question I asked and thus deserves the green check.
– Dave
Commented Oct 25, 2022 at 11:47

If we dissect AIC formula by Akaike (1974), $$AIC=-2\ln(L)+2k,$$ where $$k$$ is the number of parameters, the two differences between $$AIC_1$$ and $$AIC_2$$ are the value of $$k$$ (negative binomial likelihood will have an extra parameter) and the $$\ln(L)$$ values. Both are discrete regression models and they are often used to compare two or more models. However, one may argue if the difference in $$AIC_1$$ and $$AIC_2$$ are significant enough to conclude one model is better than the other. In this case, one may employ a likelihood ratio test and decide which model is the best.

• Sure, we can say which number is bigger, but what does that mean when the L values have different meanings? For instance, do we conclude that one model does a better job of fitting its likelihood than the other model? It still isn't clear to me what that would mean when different likelihoods are being fit.
– Dave
Commented Jul 26, 2022 at 16:57
• We can say that for sure. It is recommended to use several tests in determining the best model to avoid confusion or biasness. Pair $AIC$ with $BIC$. $BIC$ imposes larger penalty than $AIC$.
– RRMT
Commented Jul 26, 2022 at 22:46
• But what does that mean when different likelihoods are being fit? If I do a better job than you of picking apples and you do a better job than me of picking oranges…
– Dave
Commented Jul 26, 2022 at 23:20
• Actually it is more like we are both picking oranges high from the ground, but I use ladder to pick them and you jump to pick them. At the end, $Poisson$ and $NB$ are like tools and our aim is to find the best model which explains the variation in the model. Just like, we compare our efficiencies to pick oranges. Our aim is still remain to pick oranges efficiently.
– RRMT
Commented Jul 27, 2022 at 0:13