2
$\begingroup$

I have a Bayesian Hierarchical model using JAGS. In order to find the best model, I have compared the DIC of two models but It's not reliable. So, I decided to calculate WAIC from JAGS. However I have a big ambiguity and i would appreciate if you help me with this issue.

  1. Based on theory, DIC is calculated using Deviance, which is equal to -2logliklihood+constant.

Jags give us a deviance, and the dimension of Deviance is equal to (Numofchains*NumberofDraws), and DIC is calculated using deviance.

In theory, deviance= -2logliklihood+constant. So the dimension should be equal to the number of samples(observations), not (Numofchains*NumberofDraws) ??

  1. In order to calculate WAIC, I need loglikelihood. Jags does not give us Loglikelihood so, via deviance i could not get a right loglikelihood because i don't know what is the constant. So, I calculate Sum(log(normalpdf(posteriors(y,mu,sigma)))),

which returns a vector of likelihoods with the dimension of number of samples.

I am confused. I don't know which one is correct. I would appreciate your help.

$\endgroup$

1 Answer 1

3
$\begingroup$

There is more than 1 definition of DIC and WAIC. Celeux et al. (2006) provide 8 variants of DIC; Gelman and Vehtari (2013) provide 2 definitions of WAIC. Other definitions exist too.

The first three variants of DIC in Celeux et al. (2006) can be expressed generally as

$$ DIC_n = 2\overline{D({\boldsymbol\theta})}-D_n $$

where $\overline{D({\boldsymbol\theta})}$ is the posterior mean deviance,

\begin{align} \overline{D({\boldsymbol\theta})} &= E_{{\boldsymbol\theta}}\left\{D({\boldsymbol\theta})|\boldsymbol{y}\right\} \\ &= E_{{\boldsymbol\theta}}\left\{-2\log p(\boldsymbol{y}|\boldsymbol{\theta})|\boldsymbol{y}\right\} \\ &= E_{{\boldsymbol\theta}}\left\{-2\log \prod_{i=1}^N{p(y_i|\boldsymbol{\theta}_i})\Bigg|\boldsymbol{y}\right\} \\ &= E_{{\boldsymbol\theta}}\left\{-2 \sum_{i=1}^N{\log p(y_i|\boldsymbol{\theta}_i})\Bigg|\boldsymbol{y}\right\} \\ &= \frac{-2}{M}\sum_{m=1}^M\sum_{i=1}^N{\log p(y_i|\theta_i^{(m)}}) \\ \end{align}

where $N$ is the [data] sample size and $M$ is the posterior sample size, and the expectation is taken over the unknown parameter(s), $\boldsymbol\theta$, and $D_n$ is the deviance, which is a function of some (plug-in) estimate of $\boldsymbol\theta$.

Using this general expression for DIC, the first three definitions in Celeux et al. (2006) are given by the following definitions of the deviance:

$$ D_1 = -2 \log p(\boldsymbol{y}|\tilde{\theta}) $$ where \begin{align} \tilde{\theta} &= E_{{\boldsymbol\theta}}\left\{{\boldsymbol\theta}|\boldsymbol{y}\right\} \\ & = \frac{1}{MN}\sum_{m=1}^M\sum_{i=1}^N{\theta_i^{(m)}} \end{align} $$ D_2 = -2 \log p(\boldsymbol{y}|\tilde{\theta}), \hspace{3em} \tilde{\theta} = \arg\max_{\boldsymbol\theta} p(\boldsymbol{\theta}|\boldsymbol{y}) $$ and $$ D_3 = -2 \log \hat{p}(\boldsymbol{y}|\boldsymbol{\theta}) $$ where \begin{align} \hat{p}(\boldsymbol{y}|\boldsymbol{\theta}) &= E_{{\boldsymbol\theta}}\left\{p({\boldsymbol\theta}|\boldsymbol{y})|\boldsymbol{y}\right\} \\ &= \frac{1}{M}\sum_{m=1}^M\prod_{i=1}^N{ p\left(y_i \bigg|\theta_i^{(m)}\right)}. \end{align}

The other 5 definitions of DIC are a bit different (really only used for missing-value models). The definition of DIC used in JAGS is different yet again. Martyn Plummer (the author of JAGS) uses the following definition, as found in the JAGS documentation:

\begin{align} DIC_{\text{JAGS}} &= \overline{D({\boldsymbol\theta})}+0.5 \text{var}\left\{D({\boldsymbol\theta})\right\}. \end{align}

This is also the definition that is used in the R package R2jags.


Now for WAIC, the two definitions provide in Gelman and Vehtari (2013) can be expressed generally as:

$$WAIC_n = 2p_{\text{WAIC}_n} - 2\text{LPPD}$$

where the LPPD is the log pointwise predictive density, defined as:

\begin{align} LPPD &= \log \prod_{i=1}^N{p_{\text{post}}(y_i)} \\ &= \log \prod_{i=1}^N{\text{E}_{\text{post}}p(y_i|\boldsymbol{\theta}_i)} \\ &= \log \prod_{i=1}^N{\text{E}_{\boldsymbol{\theta}}\left\{p(y_i|\boldsymbol{\theta}_i)|y_i\right\}} \\ &\approx \log \prod_{i=1}^N\frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right) \\ &= \sum_{i=1}^N \log\frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right) \end{align}

The two variants of the $p_{\text{WAIC}}$ are defined as:

\begin{align} p_{\text{WAIC}_1} &= 2\sum_{i=1}^N \left\{ \log \text{E}_{\text{post}}\left[p(y_i|\boldsymbol\theta_i)\right] - \text{E}_{\text{post}}\left[\log p(y_i|\boldsymbol\theta_i\right]\right\} \\ &\approx 2\sum_{i=1}^N \left\{ \log \frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right) - \frac{1}{M} \sum_{m=1}^M \log p\left(y_i\bigg|\theta_i^{(m)}\right)\right\} \end{align}

and

\begin{align} p_{\text{WAIC}_2} &= \sum_{i=1}^N \text{Var}_{\text{post}}\log p(y_i|\boldsymbol\theta_i) \\ &= \sum_{i=1}^N \text{Var}\left[\log p(y_i|\boldsymbol\theta_i)|y_i\right] \\ &\approx \sum_{i=1}^N \frac{1}{M-1}\sum_{m=1}^M \left[\log p\left(y_i\bigg|\theta_i^{(m)}\right)-\frac{1}{M}\sum_{m=1}^M \log p\left(y_i\bigg|\theta_i^{(m)}\right)\right]^2 \\ \end{align}

Therefore, the two variants of WAIC are:

\begin{align} WAIC_1 &= 2p_{\text{WAIC}_1} - 2\text{LPPD} \\ &\approx 4\sum_{i=1}^N \left\{ \log \frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right) - \frac{1}{M} \sum_{m=1}^M \log p\left(y_i\bigg|\theta_i^{(m)}\right)\right\} - 2\sum_{i=1}^N \log\frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right) \\ &= 2\sum_{i=1}^N \left\{ \log \frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right) - \frac{2}{M} \sum_{m=1}^M \log p\left(y_i\bigg|\theta_i^{(m)}\right)\right\} \\ &= 2\sum_{i=1}^N \left\{\log \frac{1}{M}\sum_{m=1}^M L_{im} - \frac{2}{M}\sum_{m=1}^M \log L_{im}\right\} \end{align}

and

\begin{align} WAIC_2 &= 2p_{\text{WAIC}_2} - 2\text{LPPD} \\ &\approx 2\sum_{i=1}^N \frac{1}{M-1}\sum_{m=1}^M \left[\log p\left(y_i\bigg|\theta_i^{(m)}\right)-\frac{1}{M}\sum_{m=1}^M \log p\left(y_i\bigg|\theta_i^{(m)}\right)\right]^2 - 2\sum_{i=1}^N \log\frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right) \\ &= 2\sum_{i=1}^N \left\{\frac{1}{M-1}\sum_{m=1}^M \left[\log p\left(y_i\bigg|\theta_i^{(m)}\right)-\frac{1}{M}\sum_{m=1}^M \log p\left(y_i\bigg|\theta_i^{(m)}\right)\right]^2 - \log\frac{1}{M}\sum_{m=1}^M p\left(y_i\bigg|\theta_i^{(m)}\right)\right\} \\ &= 2\sum_{i=1}^N \left\{\text{var}[\log L_{im}] - \log \frac{1}{M}\sum_{m=1}^M L_{im}\right\} \end{align}

where $L_{im} = p\left(y_i\big|\theta_i^{(m)}\right)$ is just shorthand for the likelihood for the $i^{th}$ observation given the $m^{th}$ parameter estimates.

Even though the formulas for WAIC look quite complicated, the actually boil down to something quite simple to program and compuate manually from a posterior sample.


In order to answer the last part to your question, "which one is correct?", realise that there is no "correct" definition. Different definitions have been proposed to try to reflect the model fit as best as possible, given the type of data (e.g. missing value models). Think of it like trying to measure the "centrality" feature of a data set. The mean (average) may not be the best way of measuring it - that is why there are also other measures like the mode and median.

Having said that, in my experience some definitions are more robust than others, that is, less susceptible to non-interpretable values like negative values (DIC and WAIC should be positive, but it isn't gauranteed). This is because the plug-in estimate for DIC, for example, may not be a good reflection of the posterior distribution as a whole.

Of the definitions provided above, I would recommend using either $DIC_3$ or $DIC_{\text{JAGS}}$ over $DIC_1$ or $DIC_2$. And I would recommend using WAIC (probably $WAIC_2$) over any definition of DIC.

$\endgroup$
1
  • $\begingroup$ I wonder about the details of calculating var(D(theta)) in the DIC_JAGS. Could you show it? $\endgroup$
    – Keith Lau
    Commented Sep 12, 2023 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.