2
$\begingroup$

I am using Stan (Hamiltonian Monte-Carlo) to run a highly paramaterized model. One of the parameters in particular has a very low effective sample size (n_eff < .10*number of retained draws), but has an acceptable R-hat value 1.0006.

In my mind this means the chains are mixing well, but the retained samples are still highly correlated from their respective previous draws.

I have been conducting simulation work, the obtained posterior mean for the parameter is accurate, although the distribution is wider than the other parameters.

Should I be concerned about the very low effective sample size?

Other than reparamaterizing my model, are there other suggestions for resolving the low effective sample size for this parameter?

Things I have tried so far:

  1. Longer burn-in
  2. Increasing adapt_delta
  3. Thinning (although it is my understanding thinning is not useful in HMC)
  4. Using conjugate prior distributions
$\endgroup$
  • 2
    $\begingroup$ This is basic but since you don't mention it, I'm asking. Did your posterior visualizations look okay? It is possible to have good Rhat with poor convergence. See: arxiv.org/abs/1903.08008 $\endgroup$ – Heteroskedastic Jim Jul 18 '19 at 20:48
  • $\begingroup$ The trace plots look pretty typical. Once I established a reasonable warm-up, things looked well mixed. $\endgroup$ – Roman_Numerals Jul 18 '19 at 21:24
2
$\begingroup$

In BDA3 on page 285, they mention that $\hat{R}$ is an estimate of "the factor by which the scale of the current distribution for [a particular univariate parameter] might be reduced if the simulations were continued in the limit $n \to \infty$." Because your $\hat{R}$ is so close to $1$, this means that it's unlikely that running the simulation any longer will be of much use.

You mention that, for a particular parameter, it appears the "distribution is wider than the other parameters." This might be a property of the marginal posterior of that parameter, which is fine. A few of your suggestions might actually be invalidating some of the theory you're using to justify your estimates. For example, I wouldn't fiddle with the burn-ins too much, and changing the priors after you don't like simulation results is generally a no-no.

Edit: Regarding your request for more information about effective sample size, close after the portion I cited above in the same book, they discuss $\hat{R}$'s connection with effective sample size $n_{\text{eff}}$ as well as recommended settings for both. In particular, they mention that it might suffice to have as few as $10$ or $100$ as an effective sample size, and only aiming for more if an increased amount of precision is desired. Here's a quote:

As a default rule, we suggest running the simulation until $\hat{n}_{\text{eff}}$ is at least $5m$... Having an effective sample size of $10$ per sequence should typically correspond to stability of all the simulated sequences.

Here, $m$ is the number of chains you're simulating. Notice that in particular that the desired number they suggest does not depend on the number of draws, retained or otherwise. The reason why these two metrics are "typically" related is because their formulas make use of the same quantities.

However, they are getting at the dispersion of two different quantities. $\hat{R}$ measures the potential reduction in scale for the posterior distribution, while effective sample size gives you the reduction in the variance of the estimate for an expected value of this distribution. In the ideal case, we wouldn't have to resort to sampling, and we could just derive, say, the posterior expected value; here, we would have zero error on the latter, but a possibly very large number for the former. For more information take a look at page 286 and 287.

$\endgroup$
  • $\begingroup$ Thank you for your response Taylor. I appreciate your discussion of R-hat. Regarding changing priors, I agree with what you say when analyzing data for typical reasons, like inference. However, I am developing a novel model, not actually analyzing data for inference. I am fiddling with every aspect of the model to try and understand when it does and does not perform well. Could you clarify what you think about my original question? Do you think the low effective sample size should be concerning for a single parameter when R-hat is acceptable? $\endgroup$ – Roman_Numerals Jul 19 '19 at 14:10
  • $\begingroup$ @Roman_Numerals oh okay sounds good. The book I cited has several pages on rules of thumb for effective sample size and Rhat. I've just edited the post to try to summarize some of the ones you might find relevant. $\endgroup$ – Taylor Jul 19 '19 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.