6
$\begingroup$

I am using Stan (Hamiltonian Monte-Carlo) to run a highly paramaterized model. One of the parameters in particular has a very low effective sample size (n_eff < .10*number of retained draws), but has an acceptable R-hat value 1.0006.

In my mind this means the chains are mixing well, but the retained samples are still highly correlated from their respective previous draws.

I have been conducting simulation work, the obtained posterior mean for the parameter is accurate, although the distribution is wider than the other parameters.

Should I be concerned about the very low effective sample size?

Other than reparamaterizing my model, are there other suggestions for resolving the low effective sample size for this parameter?

Things I have tried so far:

  1. Longer burn-in
  2. Increasing adapt_delta
  3. Thinning (although it is my understanding thinning is not useful in HMC)
  4. Using conjugate prior distributions
$\endgroup$
2
  • 2
    $\begingroup$ This is basic but since you don't mention it, I'm asking. Did your posterior visualizations look okay? It is possible to have good Rhat with poor convergence. See: arxiv.org/abs/1903.08008 $\endgroup$ Jul 18, 2019 at 20:48
  • $\begingroup$ The trace plots look pretty typical. Once I established a reasonable warm-up, things looked well mixed. $\endgroup$ Jul 18, 2019 at 21:24

1 Answer 1

7
$\begingroup$

In BDA3 on page 285, they mention that $\hat{R}$ is an estimate of "the factor by which the scale of the current distribution for [a particular univariate parameter] might be reduced if the simulations were continued in the limit $n \to \infty$." Because your $\hat{R}$ is so close to $1$, this means that it's unlikely that running the simulation any longer will be of much use.

You mention that, for a particular parameter, it appears the "distribution is wider than the other parameters." This might be a property of the marginal posterior of that parameter, which is fine. A few of your suggestions might actually be invalidating some of the theory you're using to justify your estimates. For example, I wouldn't fiddle with the burn-ins too much, and changing the priors after you don't like simulation results is generally a no-no.

Edit: Regarding your request for more information about effective sample size, close after the portion I cited above in the same book, they discuss $\hat{R}$'s connection with effective sample size $n_{\text{eff}}$ as well as recommended settings for both. In particular, they mention that it might suffice to have as few as $10$ or $100$ as an effective sample size, and only aiming for more if an increased amount of precision is desired. Here's a quote:

As a default rule, we suggest running the simulation until $\hat{n}_{\text{eff}}$ is at least $5m$... Having an effective sample size of $10$ per sequence should typically correspond to stability of all the simulated sequences.

Here, $m$ is the "number of chains (after splitting)" BDA3 pg. 284 you're simulating . Notice that in particular that the desired number they suggest does not depend on the number of draws, retained or otherwise. The reason why these two metrics are "typically" related is because their formulas make use of the same quantities.

However, they are getting at the dispersion of two different quantities. $\hat{R}$ measures the potential reduction in scale for the posterior distribution, while effective sample size gives you the reduction in the variance of the estimate for an expected value of this distribution. In the ideal case, we wouldn't have to resort to sampling, and we could just derive, say, the posterior expected value; here, we would have zero error on the latter, but a possibly very large number for the former. For more information take a look at page 286 and 287.

$\endgroup$
4
  • $\begingroup$ Thank you for your response Taylor. I appreciate your discussion of R-hat. Regarding changing priors, I agree with what you say when analyzing data for typical reasons, like inference. However, I am developing a novel model, not actually analyzing data for inference. I am fiddling with every aspect of the model to try and understand when it does and does not perform well. Could you clarify what you think about my original question? Do you think the low effective sample size should be concerning for a single parameter when R-hat is acceptable? $\endgroup$ Jul 19, 2019 at 14:10
  • $\begingroup$ @Roman_Numerals oh okay sounds good. The book I cited has several pages on rules of thumb for effective sample size and Rhat. I've just edited the post to try to summarize some of the ones you might find relevant. $\endgroup$
    – Taylor
    Jul 19, 2019 at 16:22
  • $\begingroup$ Thanks for the nice explanation. Helpful. Just a comment, here m is twice the number of chains we are simulating (according to BDA3, p-287). $\endgroup$
    – Dey
    Aug 3, 2020 at 12:17
  • $\begingroup$ @Dey From my read of BDA3, m is twice the number of chains if you follow the approach of BDA3 authors and split each chain in two to estimate mixing. In contrast, I think this is most commonly handled in the MCMC packages I use (PyMC3, Numpyro) with the user specifying the number of chains when configuring MCMC, not sub-dividing the independent MCMC traces after simulation before calculating these statistics. $\endgroup$
    – David Diaz
    Jan 31 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.