# Preventing Pareto smoothed importance sampling (PSIS-LOO) from failing

I recently started using Pareto smoothed importance sampling leave-one-out cross-validation (PSIS-LOO), described in these papers:

• Vehtari, A., & Gelman, A. (2015). Pareto smoothed importance sampling. arXiv preprint (link).
• Vehtari, A., Gelman, A., & Gabry, J. (2016). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. arXiv preprint (link)

This represents a very enticing approach to out-of-sample model evaluation as it allows to perform LOO-CV with a single MCMC run, and it is allegedly better than existing information criteria such as WAIC.

PSIS-LOO has a diagnostics to tell you whether the approximation is reliable, namely given by the estimated exponents $\hat{k}_i$ of the Pareto distributions fitted to the tails of the empirical distributions of importance weigths (one weight per data point). In short, if an estimated weight $\hat{k}_i \gtrsim 0.7$, bad things can happen.

Sadly, I found that in my application of this method to my problem, for the majority of models of interest I find that a large fraction of the $\hat{k}_i \gg 0.7$. Unsurprisingly, some of the reported LOO log-likelihoods were quite obviously nonsensical (compared to other datasets). As a double-check, I performed a traditional (and time consuming) 10-fold cross-validation, finding that indeed in the above case PSIS-LOO was giving awfully wrong results (on the upside, results were in very good agreement with 10-fold CV for the models in which all $\hat{k}_i \ll 0.7$). For the record, I am using the MATLAB implementation of PSIS-LOO by Aki Vehtari.

Maybe I am just very unlucky in that my current and first problem in which I apply this method is "difficult" for PSIS-LOO, but I suspect that this case might be relatively common. For cases such as mine, the Vehtary, Gelman & Gabry paper simply says:

Even if the PSIS estimate has a finite variance, when $\hat{k} > 0.7$, the user should consider sampling directly from $p(\theta^s |y_{−i})$ for the problematic $i$, use $k$-fold cross-validation, or use a more robust model.

These are obvious but not really ideal solutions as they are all time consuming or require additional fiddling (I appreciate that MCMC and model evaluation are all about fiddling, but the less the better).

Is there any general method that we can apply beforehand to try and prevent PSIS-LOO from failing? I have a few tentative ideas, but I wonder if there is already an empirical solution that people have been adopting.