What are the different ESS (Effective Sample Size) methods in arviz?

Question

I am running a Markov Chain Monte Carlo sampler for phylogenetic inference. I used to track the progress of convergence by checking the Effective Sample Size in Tracer. Because the results started to look good, I have begun to implement some analytics in Python, and while at it I also added ArviZ ESS calculation.

For most of my chains and most of my parameters, ArviZ and Tracer agree concerning the order of magnitude of the Effective Sample Size, but there are a few runs that have all parameters with ESS in the several hundreds according to Tracer, but very low ESS according to ArviZ.

I noticed that ArviZ exposes several different methods of computing ESS, but I cannot find documentation what they mean. (The R package Rhat also provides ‘bulk’ and ‘tail’, so I get a vague idea what they do.

What do the other options mean? Do you have any advice on which methods I should use for what purpose?

Answer 1

I was wondering the same thing just a few weeks ago. I don't have the definitive answer, but a few thoughts.

ArviZ cites https://arxiv.org/abs/1903.08008 in which the authors discuss more robust PSRF variants and also ESS calculations. They also discuss a variant in case you're more interested in rare events; These rare events are in the tail of the distribution.

Furthermore the paper has some useful rules of thumb, such as running 4 individual chains, which should reach an ESS total of 400 (50 per chain after the chains are split).

The Stan community also talks about the fact, that ESS is undererstimated, when the chains have not yet converged. https://mc-stan.org/docs/2_19/reference-manual/effective-sample-size-section.html

I could not find how you calculate ESS in tracer after a short look. One scenario might be, that tracer calculates the ESS of a single chain without splitting it into two parts. Because ArviZ splits the ESS chains into two parts, it should/could notice, if the chain has not converged yet and therefore undererstimate the ESS.

I would use 4 parallel chains and trust the ArviZ ESS estimation. The ESS targets from the Vhetari paper work well for my use-cases. Note that the ESS estimate will only be correct, if the chain has converged anyways.