I have read in Bayesian Data Analysis by Andrew Gelman that log predictive density can be used to compare Bayesian models due to its connection to the Kullback-Leibler information.

The log predictive density has an important role in statistical model comparison because of its connection to the Kullback-Leibler information measure. In the limit of large sample sizes, the model with the lowest Kullback-Leibler information—and thus, the highest expected log predictive density—will have the highest posterior probability. Thus, it seems reasonable to use expected log predictive density as a measure of overall model fit. Due to its generality, we use the log predictive density to measure predictive accuracy in this chapter.

The Kullback-Leibler divergence between $Q(\theta)$ and $P(\theta|y)$ is,

$D_{KL}(Q||P) = \sum_{\theta}Q(\theta)ln\frac{Q(\theta)}{P(\theta|y)}$

$D_{KL}(Q||P) = E_\theta[lnQ(\theta)] -E_\theta[lnP(\theta,y)]+lnP(y)$

$D_{KL}(Q||P) = E_\theta[lnQ(\theta)] -E_\theta[lnP(y|\theta)P(\theta)] +lnP(y)$

  1. Is this the connection between KL information and the expected log predictive density?
  2. If this is the connection, how do we assume that maximum expected log predictive density implies the minimum KL divergence. Comparing different models means changing the values for $\theta$. However, when changing $\theta$ the term $E_\theta[lnQ(\theta)]$ also changes. Why do we neglect the effect of this term to the KL divergence ?
  3. Is not the predictive density also used to calculate the prediction for unseen $\bar{y}$ ? The book also says,

The ideal measure of a model’s fit would be its out-of-sample predictive performance for new data produced from the true data-generating process (external validation).

Therefore, choosing the model with maximum predictive density for new data points means that we choose the model that gives maximum value for the predictions. How could a model gives maximum value for the prediction is better than other models?

I appreciate if someone can help me to understand these connections. Thanks



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.