Say I have some Bayesian predictive model I assume to be true for each observation $x_1$. Each $x_2$ is a latent/unseen/hidden random variable. The parameters are $\theta$. It's a mixture distribution: $$ p(x_1) = \int p(x_1|x_2,\theta)p(x_2|\theta)p(\theta)dx_2d\theta. $$

I can sample, for $i=1,\ldots,N$, $\theta^i \sim p(\theta)$, then sample $X_2^i \sim p(x_2|\theta^i)$, and then have the approximation \begin{equation} \hat{p}_1(x_1) = \frac{1}{N}\sum_{i=1}^N p(x_1|X_2^i,\theta^i). \end{equation}

Now, assume I cannot practically do this, and rather I either a) decide to sample the $\theta^i$ portion only once and re-use that, ignoring the uncertainty about it $p(\theta)$, or b) I just use the wrong distribution, say $q(\theta)$. So $\theta^*$ will either be a mean, median, mode or sample from $p()$, or it will be some summary from $q()$. I would be using the less accurate approximation
$$ \hat{p}_2(x_1) = \frac{1}{N}\sum_{i=1}^N p(x_1|\tilde{X}_2^i,\theta^*). $$


How do I measure how far off $\hat{p}_2(\cdot)$ is from $\hat{p}_1(\cdot)$? Kullback-Leibler divergence? Anything else?


I have heard of "scoring rules" that evaluate the predictive ability of distributions by looking at how well they line up with the observed data. This is a predictive model I have here, so I am concerned with how much worse off $\hat{p}_2$ does if I have to use it, instead of $\hat{p}_1$.

For example, using the above notation, the "logarithmic scoring rule" takes an observation $X_1$ and evaluates $\hat{p}_i(\cdot)$'s predictive ability by computing $\log \hat{p}_i(X_1)$.

It seems that there are some connections between the logarithmic scoring rule, and KL-divergence. For example KL-distance \begin{align*} KL(\hat{p}_1||\hat{p}_2) &= E_{\hat{p}_1}[-\log\hat{p}_2(X_1)] - E_{\hat{p}_1}[-\log\hat{p}_1(X_1)] \\ &= E_{\hat{p}_1}[\log\hat{p}_1(X_1)] - E_{\hat{p}_1}[\log\hat{p}_2(X_1)], \end{align*} is the difference between the cross entropy and the entropy of the "true" density, but more to the point here, can also be seen as the reduction in average score if you're using the "wrong" model, $\hat{p}_2$.

This isn't a perfect example, though, because we're averaging over the left side variable down here, and this isn't what's going on above. Ultimately I am looking for some procedure that a) tells me whether this approximation is okay to do (tells me not to worry when there won't be much reduction in some meaningful quantity related to predictive power), and b) suggests some guidelines for how to select $\theta^*$. Should I take the mean, median or mode of $p(\theta)$?

I do not know much about these scoring rules. Are there other ways to measure predictive ability? What are the most common ways? Which distance will be strongly tied to a highly-used metric of predictive power?

  • $\begingroup$ There is no answer to your question as stated in that every functional distance can be used: Kullback, Hellinger, $L_2$, &tc. You should thus specify in which sense you want to pick a particular distance, i.e., what are your criteria for preferring a functional distance to the next. $\endgroup$ – Xi'an Jan 31 '17 at 7:17
  • 1
    $\begingroup$ @Xi'an sorry for the vagueness. I'm not even sure if it's answerable yet. It's still sort of vague. It's funny you should respond to this because it was one of your papers that motivated this question. The example I am thinking of is of online learning for state space models. In this example $p(\theta)$ is some possibly outdated posterior from a batch mcmc algorithm. My goal is filtering for predicting out of sample data. I'm wondering how to use that posterior for predicting, and if it's okay to use an outdated posterior. $\endgroup$ – Taylor Jan 31 '17 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.