Can WAIC be used to compare Bayesian linear regression models with different likelihoods?

Question

I would like to use WAIC to help with model selection, where the models are simple linear regressions with Bayesian inference, non-flat priors and MCMC estimation.

I am currently considering two such linear models, both of which have the same dependent variables and (Normal) priors for the regression coefficients, but one has a Normal likelihood and the other has a Student-T likelihood (i.e. for robust regression). I appreciate that information criteria such as WAIC are not the be-all-and-end-all of the model selection process, but I was intending to use WAIC as part of this analysis.

However, when I turned to Chapter 9 of ‘Statistical Rethinking’ by Richard McElreath, he says that,

“… it is tempting to use information criteria to compare models with different likelihood functions… Unfortunately, WAIC (or any other information criterion) cannot sort it out. The problem is that deviance is part normalising constant. The constant affects the absolute magnitude of the deviance, but it doesn’t affect fit to data. Since information criteria are all based on deviance, their magnitude also depends on these constants. That is fine, so long as all of the models you compare use the same outcome distribution type… In that case, the constants subtract out when you compare models by their differences. But if the two models have different outcome distributions, the constants don’t subtract out and you can misled by a difference in AIC/DIC/WAIC”

My problem, is that I cannot see or derive this result (these constants), from the definition of the deviance given in the same book,

$$ D(q) = -2 \sum_{i} log(q_i) $$

where i indexes each observation and $q_{i}$ is just the likelihood of case i.

Now, I appreciate that the deviance is intended as an approximation to the cross-entropy term in the Kullback-Leibler (KL) divergence between two distributions - e.g. p for the ’true’ distribution of the data and q for the distribution implied by my model,

$$ D_{KL} = E_{p}[log(p)] - E[log(q)] $$

where the cross-entropy, and thereby the deviance, represent an attempt to measure the additional entropy introduced (or the information lost) by using distribution q to describe distribution p.

I can see that we cannot know $E_{p}[log(p)]$, which is a constant for all models being compared, but that this constant term disappears when making relative comparisons for some estimate of $E[log(q)]$ - i.e. when making relative comparisons of the deviance. I would expect this to be true even when we are comparing models using different likelihoods, as p is the ‘true’ distribution of the data and not related to the likelihood of the models in any way - is this correct?

So, in summary, can I use WAIC to compare models with different likelihood functions, and if not, why not?

Answer 1

For your specific case, I would point out that the Student's t includes the normal distribution as a limiting case (df -> inf). Thus, the two are nested and not really different likelihoods. Because of that, I don't really see a need for model selection - you can just fit the Student's t and interpret the df value as proximity to normality. If you are super concerned about overfitting, add a regularization (hyper)prior on the df parameter. Note that it may be useful to reparameterize the df parameter in the Student's t, see, e.g. Augustynczik et al. (2017) Forest Ecology and Management, 401, 192-206.

In general: yes, you can compare different likelihoods with ICs such as AIC or WAIC, with exceptions. These exceptions are probably the thought underlying the quoted paragraph, but I admit that the text is sufficiently vague to create confusion.

Generally, different likelihoods are comparable (note, btw., that the use of deviance in the text is a bit confusing because deviance is often defined as the difference to a saturated model, but here only means log L). However, there are a number of exceptions. Some common situations are

1. Changes of the # of data points
2. Changing the scale of the response variable (e.g. doing a log transformation on y), see here
3. Changing the codomain of the probability distribution, e.g. comparing continuous with discrete distributions

I think 1 is trivial (and easy to correct for). For 2,3 consider that p(D|M, parameters) is a pdf for D, thus changing the scale or codomain will change the integral and thus the normalized density. See also related questions on CV, e.g. here. Another problem could be that you use a stats software that doesn't use properly normalized likelihood values (usually the normalization is not important, so a programmer might be tempted to drop it), but I don't think that's common.

Answer 2

Since you ask about WAIC, it helps if we forget deviance and focus on (negative) log score as in WAIC paper.

First to make the terms more clear, p(y|theta) as a function of y is an observation model and p(y|theta) as a function of theta is a likelihood. WAIC and LOO (usually) focus on (log) predictive density in continuous case or probability in discrete case. Your question would make more sense if you would ask "whether WAIC (or LOO) can be used to compare model with different observation models ?"

• You can compare models given different discrete observation models and it's also allowed to have different y as long as the mapping is bijective.
• You can't mix densities and probabilities, so you can't compare model given continuous and discrete observation models, unless you compute probabilities in intervals from the continuous model
• You can compare models given different continuous observation models and but you have to have exactly the same y. If y is transformed, then the Jacobian of that transformation needs to be included.

I repeat that you need to focus on the observation model. As a bonus there is at least one case where a discrete observation model and a continuous observation model have the same continuous likelihood, so knowing the likelihood doesn't help deciding whether the two models can be compared.