5
$\begingroup$

I have three predictive models (regressions) which parameters are estimated by Markov Chain Monte Carlo. Predictions are made over a test set of size $N$. Since I compare the models under different settings, I would like to summarize the results in a table.

I have considered using Means Squared Error (MSE) or $R^2$ but I wonder if I could be fully bayesian and use all the posterior sampled by MCMC, and not just the means of the predictions.

For instance, MSE would be something like: $$ \text{MSE} = \frac{1}{N} \sum_{i=1}^N (y_i^{true} - \mathbb{E}[y_i^{MC}])^2 $$

where $\mathbb{E}[y_i^{MC}]$ is the average of the $y_i$ samples.

However, it seems reasonable to me that a tight posterior around the good prediction should have a better score than a flatter posterior. And yet, I don't find any reference to this kind of method.

Questions:

  • What is the/a correct bayesian way to do it? Does it have a name?

  • What is the name for techniques that check the accuracy of posteriors-based predictions in test sets where we know the true outputs? [1]

[1] : I say that because I've realized that a lot of bayesian literature talks about assessing the quality of a prediction (e.g.: the outcome of the elections) before we know the real outcome.

Edit:

What about taking the mean of the mean squared errors? That is averaging the squared errors for the samples at every test point, and then averaging the results through all the points.

\begin{align} \frac{1}{N}\sum_{i=1}^N\left[\frac{1}{M}\sum_{j=1}^M\left(y_i^{true}- y_i^{(j)}\right)^2\right] \end{align} where $M$ is the number of samples from the MCMC and $y_i^{(j)}$ is the prediction, for the input $x_i$, using the $j$-th sample of the model parameters $\boldsymbol{\theta^{(j)}}$.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ This looks like you should be use the posterior variance - this is what you would be using in the second formula. Beware of using highly auto-correlated MCMC chains to do this though, as it will make the variance seem smaller. $\endgroup$
    – probabilityislogic
    Commented Feb 28, 2015 at 22:27
  • $\begingroup$ Sorry! There was a mistake in the third formula. I meant the squared difference between the posterior prediction and the true value. Do you mean that I should report both the MSE and the posterior variance? $\endgroup$
    – alberto
    Commented Feb 28, 2015 at 22:35
  • 1
    $\begingroup$ The average posterior variance for each data point is the posterior expectation of what you have called MSE. $\endgroup$
    – probabilityislogic
    Commented Mar 1, 2015 at 0:08
  • $\begingroup$ I just found this survey projecteuclid.org/download/pdfview_1/euclid.ssu/1356628931 and discovered the "L-criterion" (page 198, eq 111). Isn't it closer (or exactly) to what I am doing in my edit? $\endgroup$
    – alberto
    Commented Mar 1, 2015 at 9:10

1 Answer 1

Reset to default
1
$\begingroup$

You want the posterior predictive distribution, i.e. for a test vector $\tilde{y}$ and training vector $y$

$$ p(\tilde{y}|y) = \int p(\tilde{y}|\theta) p(\theta|y) d\theta). $$

As you point out, if this distribution is tighter around the test data $\tilde{y}$, then it will give a higher density.

If you only have samples from the posterior, then you can approximate this via

$$ p(\tilde{y}|y) \approx \frac{1}{M} \sum_{j=1}^M p(\tilde{y}|\theta^{(j)}) $$

where $\theta^{(j)} \sim p(\theta|y)$.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I was re-reading Gelman's Bayesian Data Analysis and I think it's exactly that ! So, to be clear $\bar{y}$ would be the observed / true / target predictions, wouldn't they? $\endgroup$
    – alberto
    Commented Mar 2, 2015 at 21:36
  • $\begingroup$ For the record, I see that if we factorized $\tilde{y}$ into the different $\tilde{y_i}$ and take the logarithmic version of that is called "log pointwise predictive density". $\endgroup$
    – alberto
    Commented Mar 3, 2015 at 8:17
  • 1
    $\begingroup$ Yes $\tilde{y}$ is the true values. Depending on the model structure, you may be able to assume the $\tilde{y}_i$ are conditionally independent and thus factorize. $\endgroup$
    – jaradniemi
    Commented Mar 3, 2015 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.