7
$\begingroup$

I've been using Pareto-smoothed importance sampling (PSIS-LOO) cross-validation for diagnosis and comparing Bayesian models fitted with Stan/brms for a while now. From my experience with using it on both synthetic and real data, it seems to perform extremely well. So, naturally, I'd like to properly understand the method. I don't come from a quantitative background, I only got into Bayesian statistics about 1 year ago and I've been playing catch-up ever since then. Anyway, I've tried studying up on importance sampling and read the original Vehtari & Gelman (2015) paper a couple times now, but I still can't really connect the dots. I feel like I have a basic grasp of importance sampling, but I still don't understand how it can be used to evaluate the posterior predictive density.

Does anyone have an intuitive explanation of PSIS-LOO? What it does and why it works? Ideally with as few formulas as possible. Thanks!

$\endgroup$
3
  • $\begingroup$ Are you asking about the estimation method or what is being estimated? $\endgroup$ Jun 14, 2020 at 0:25
  • 2
    $\begingroup$ The estimation method, i.e. how we get expected log predictive density via importance sampling, without having to refit the model. $\endgroup$
    – Adam B.
    Jun 14, 2020 at 21:38
  • $\begingroup$ Preprint link for the Vehtari & Gelman paper - arxiv.org/abs/1507.04544?context=stat $\endgroup$
    – JMzance
    Aug 2, 2022 at 17:07

1 Answer 1

5
$\begingroup$

I'm a scientist, not a statistician, but given that it has been almost a year since this question was first asked and it still has not received any answers, I will do my best to provide some insight. Revisions/corrections to my answer are welcome as I have also spent considerable time and effort trying to figure out this paper. I'll mention here that I too have been pleased by this method's performance and will encourage anyone reading this answer to give it a shot if you have not already.

To start from the top: importance sampling is useful when you are interested in a probability distribution $f(x)$ but are unable to sample from it using MCMC due to computational and/or practical constraints. Fortunately, you can instead sample from an approximating distribution $g(x)$ and use those samples to make inferences about $f(x)$. To do so, simply adjust the weight of each sample to $f(x)/g(x)$. In principle, this adjustment corrects both 1) the overrepresentation of samples that are more prevalent in the sampling distribution $g(x)$ than in the target distribution $f(x)$, and 2) the underrepresentation of samples that are rare in $g(x)$ but common in $f(x)$.

The problems that are commonly described about importance sampling arise from this second set of samples. Depending on the overlap between the sampling distribution and the target distribution, cases that are rare in $g(x)$ but common in $f(x)$ will have huge importance weights and will strongly influence any inferences you make about the composition of the target distribution $f(x)$. Often, the tails of the resulting distribution are erratic. Based on my understanding, the primary contribution of the PSIS paper is to observe that these weights follow the Pareto principle, sometimes called the "80/20 rule" (although the numbers in this case are arbitrary). Therefore, rather than take these weights at face value, the authors demonstrate that you can fit them to a generalized Pareto distribution, thus smoothing over their values and removing many of these issues observed with importance sampling in general.

So what does this have to do with leave-one-out cross-validation (LOO-CV)? To recap LOO-CV, you are typically interested in gathering uncertainty statistics about your model by iterating through your dataset, removing a data point, and repeating the analysis (e.g. MCMC sampling). Most of us would describe the tediousness of this process, namely the resampling step, as a "computational and/or practical constraint", and the authors of this paper appear to agree. Their solution is to skip resampling entirely and instead use PSIS to reweigh the samples obtained the first time with the complete dataset, thus saving considerable amounts of time while still providing the uncertainty statistics of interest. As they demonstrate, this provides results that are nearly identical to classical LOO-CV, but with a far lower computational footprint.

To summarize PSIS-LOO:

  1. Use the complete data set to generate a large pool of samples from the sampling distribution $g(x)$.
  2. Iterate through each data point and remove it from the data set.
  3. Use Pareto-smoothed importance sampling to reweigh the samples obtained in step one to this new target distribution $f(x)$.
  4. After repeating for the full dataset, compute the parameter statistics using LOO-CV as you previously would.

Again, if you are a statistician reading this and you are thinking "this answer is wrong", then I would appreciate any corrections, revisions, or addendums.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.