Preventing Pareto smoothed importance sampling (PSIS-LOO) from failing

Question

I recently started using Pareto smoothed importance sampling leave-one-out cross-validation (PSIS-LOO), described in these papers:

• Vehtari, A., & Gelman, A. (2015). Pareto smoothed importance sampling. arXiv preprint (link).
• Vehtari, A., Gelman, A., & Gabry, J. (2016). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. arXiv preprint (link)

This represents a very enticing approach to out-of-sample model evaluation as it allows to perform LOO-CV with a single MCMC run, and it is allegedly better than existing information criteria such as WAIC.

PSIS-LOO has a diagnostics to tell you whether the approximation is reliable, namely given by the estimated exponents $\hat{k}_i$ of the Pareto distributions fitted to the tails of the empirical distributions of importance weigths (one weight per data point). In short, if an estimated weight $\hat{k}_i \gtrsim 0.7$, bad things can happen.

Sadly, I found that in my application of this method to my problem, for the majority of models of interest I find that a large fraction of the $\hat{k}_i \gg 0.7$. Unsurprisingly, some of the reported LOO log-likelihoods were quite obviously nonsensical (compared to other datasets). As a double-check, I performed a traditional (and time consuming) 10-fold cross-validation, finding that indeed in the above case PSIS-LOO was giving awfully wrong results (on the upside, results were in very good agreement with 10-fold CV for the models in which all $\hat{k}_i \ll 0.7$). For the record, I am using the MATLAB implementation of PSIS-LOO by Aki Vehtari.

Maybe I am just very unlucky in that my current and first problem in which I apply this method is "difficult" for PSIS-LOO, but I suspect that this case might be relatively common. For cases such as mine, the Vehtary, Gelman & Gabry paper simply says:

Even if the PSIS estimate has a finite variance, when $\hat{k} > 0.7$, the user should consider sampling directly from $p(\theta^s |y_{−i})$ for the problematic $i$, use $k$-fold cross-validation, or use a more robust model.

These are obvious but not really ideal solutions as they are all time consuming or require additional fiddling (I appreciate that MCMC and model evaluation are all about fiddling, but the less the better).

Is there any general method that we can apply beforehand to try and prevent PSIS-LOO from failing? I have a few tentative ideas, but I wonder if there is already an empirical solution that people have been adopting.

Answer 1

For the record, I posted a similar question to the Stan users mailing list, which you can find here. I was answered by one of the authors of the original PSIS-LOO paper and by other contributors of Stan. What follows is my personal summary.

The short answer is that there are no known general methods to prevent PSIS-LOO from failing. If PSIS-LOO fails, it is usually because the model has issues, and fixing it is necessarily left to the user.

Specifically, the reason why PSIS-LOO may fail is usually because one or more LOO distributions are shifted and/or broader than the full posterior, likely due to influential observations, and the importance sampling distribution collapses to one or a few points.

I was thinking that you could try to adopt some form of parallel posterior tempering approach to solve this issue. The idea is not necessarily wrong, but it was pointed out to me that:

• textbook posterior tempering would still require a lot of case-by-case fiddling to find the right temperature level(s), as there is no obvious nor known way to do that (incidentally, for this reason Stan does not include parallel tempering);
• if you use more than two temperature levels (as it may be required to have a robust approach), the final computational cost approaches that of K-fold cross validation, or of running MCMC on the problematic LOO distributions.

In short, if PSIS-LOO fails, it seems to be hard to get a method that is as robust and general as other simple patches; that's why Vehtari, Gelman & Gabry suggested those methods as per the quote I posted in my original question.