0
$\begingroup$

The PSIS-LOO information criterion (see this paper by Vehtari, Gelman, and Gabry) assigns a Pareto shape parameter $\hat k$ to each observation in the data, and these $\hat k$ values can be used to assess the reliability of the estimates.

According to the authors:

Even if the PSIS estimate has a finite variance, when $\hat{k} > 0.7$ the user should consider sampling directly from $p(\theta^s |y_{−i})$ for the problematic $i$, use K-fold cross-validation, or use a more robust model.

The first option, sampling directly from $p(\theta^s |y_{−i})$ for the problematic $i$, has been implemented in the R packages rstanarm and brms.

This earlier paper by Gelman, Hwang, and Vehtari states that, to correct for the left-one-out fits being conditioned on one less observation, one can add a first-order bias correction $b$ to the loo cross validation estimate (see section 3.8).

Main question: How should we include $b$ when replacing bad PSIS-LOO estimates with explicit loo cross validations?

In the notation of the Gelman, Hwang, and Vehtari paper, rstanarm and brms both replace a bad PSIS-LOO estimate for row $i$ with $\log p_{\text{post}(-i)}(y_i)$. Based on the definition for $b$, it seems like the bias-corrected version of this could be something like $\log p_{\text{post}(-i)}(y_i) + b_i$, where

$$ b_i = \log p_{\text{post}}(y_i) - \frac{1}{n} \sum_{j=1}^{n} \log p_{\text{post}(-i)}(y_j). $$

Does this seem right? In tests, adding this bias correction seems to always increase the resulting looic estimate, but I'm not sure if this is just a quirk of the particular data I'm using.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.