12
$\begingroup$

I want to calculate the log pointwise predictive density from an MCMC sample. Gelman et al (2014) define the lppd as

$$ lppd = \sum_i^n \log \int( p(y_i| \theta) p_\text{post}(\theta) d\theta $$

My questions are

  1. Why is the lppd called "point-wise" if we sum over all points data points?
  2. Am I correct in interpreting this as lppd being identical to the marginal likelihood, except for the fact that we marginalize the likelihood over the posterior distribution instead of the prior distribution? Again, I find the name a bit misleading, something like marginal posterior likelihood would be more intuitive to me.
  3. It seems to me that the easiest way to calculate the lppd in practice would be to record the likelihood in the MCMC, and then take the mean of these values. Except for the memory used for recording the likelihood, is there any drawback to this approach?

Edit: I just wanted to note that after the answer of @PITBULL, I realize now that the key point is that a kind of marginal posterior likelihood is calculated per data point first, and then those probabilities are multiplied, so I do understand the name choice now, and the answer to the next two questions is "no" and "no" ;)

References:

Gelman, A.; Hwang, J. & Vehtari, A. (2014) Understanding predictive information criteria for Bayesian models. Stat. Comput., Springer US, 24, 997-1016.

$\endgroup$

1 Answer 1

7
$\begingroup$
  1. Pointwise because you are calculating predictive density values for each point observation. Note that you could take 2 or 3 observations together instead. At the end of the day, it is just a name that Gelman invented for marketing purposes.
  2. No, it is not the (log)marginal likelihood. In the marginal likelihood you integrate the full likelihood multiplied by the prior, with respect to the parameters (and the integral of the logarithm is not the logarithm of the integral). Note that the lppd is larger when the samples are close to the mode(s) or in high density regions, which is what you would like to achieve typically with a statistical model.
  3. If you have a posterior sample, calculating the lppd is straightforward since you just need to plug in the samples in a Monte Carlo integration:

$$ \int p(y_i| \theta) p_\text{post}(\theta) d\theta \approx \dfrac{1}{S}\sum_{s=1}^S p(y_i\vert \theta^{(s)})$$

$\endgroup$
1
  • $\begingroup$ The indices on the summation are wrong, no? The index of the data point should be different from the index of the sample of the parameter. Gelman uses $y_i$ and $\theta^s$ in the definition to differentiate them (p.5). $\endgroup$
    – Kuku
    May 2, 2022 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.