The PSIS-LOO information criterion (see this paper by Vehtari, Gelman, and Gabry) assigns a Pareto shape parameter $\hat k$ to each observation in the data, and these $\hat k$ values can be used to assess the reliability of the estimates.

According to the authors:

Even if the PSIS estimate has a finite variance, when $\hat{k} > 0.7$ the user should consider sampling directly from $p(\theta^s |y_{−i})$ for the problematic $i$, use K-fold cross-validation, or use a more robust model.

The first option, sampling directly from $p(\theta^s |y_{−i})$ for the problematic $i$, has been implemented in the R packages rstanarm and brms.

This earlier paper by Gelman, Hwang, and Vehtari states that, to correct for the left-one-out fits being conditioned on one less observation, one can add a first-order bias correction $b$ to the loo cross validation estimate (see section 3.8).

Main question: How should we include $b$ when replacing bad PSIS-LOO estimates with explicit loo cross validations?

In the notation of the Gelman, Hwang, and Vehtari paper, rstanarm and brms both replace a bad PSIS-LOO estimate for row $i$ with $\log p_{\text{post}(-i)}(y_i)$. Based on the definition for $b$, it seems like the bias-corrected version of this could be something like $\log p_{\text{post}(-i)}(y_i) + b_i$, where

$$ b_i = \log p_{\text{post}}(y_i) - \frac{1}{n} \sum_{j=1}^{n} \log p_{\text{post}(-i)}(y_j). $$

Does this seem right? In tests, adding this bias correction seems to always increase the resulting looic estimate, but I'm not sure if this is just a quirk of the particular data I'm using.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.