I used Metropolis-Hastings algorithm to run a MCMC to some failure data in Bayesian analysis and the posterior samples of the parameters were obtained. There's a strong correlation between the posterior samples. Does the correlation affect the point estimates from the posterior values? Can anyone please help me with the interpretation of the correlation between the posterior parameter values?

Thank you

  • 1
    $\begingroup$ Are you talking about a correlation between consecutive samples of the same parameter, or between concurrent samples of different parameters? $\endgroup$ – Ruben van Bergen Jan 20 '17 at 10:37
  • $\begingroup$ It's about the correlation between concurrent samples of different parameters. $\endgroup$ – Nayomi Jan 23 '17 at 0:55

You clarified that you're talking about a correlation between concurrent samples of different parameters. This is different from the serial correlation that the other answer goes into (which is why I asked). Serial correlation between samples of the same parameter arises in many MCMC algorithms, because of the "random walk" behavior of the algorithm, which means that each new sample is a finite step away from the current sample. But this random walk behavior does not cause the kind of correlation you're seeing, between concurrent samples of different parameters.

A correlation between samples of different parameters normally just means that the posterior distributions of those parameters are in fact correlated. E.g. say you have some data $y$ that is bivariate Normally distributed: $$ y\sim N\left(\begin{bmatrix} \mu_1\\ \mu_2 \end{bmatrix},\begin{bmatrix} \sigma_1^2 & \rho\sigma_1\sigma_2\\ \rho\sigma_1\sigma_2 & \sigma_2^2 \end{bmatrix} \right) $$

Then the posterior $p\left(\begin{bmatrix} \mu_1\\ \mu_2 \end{bmatrix}\mid y\right)$ will be correlated (between $\mu_1$ and $\mu_2$) in proportion to $\rho$, and therefore samples of $\boldsymbol{\mu}$ under this posterior will also be correlated with each other.

So unless you have reason to believe that the posteriors shouldn't be correlated, there is nothing to worry about (and to answer your question: your interpretation of this correlation should be that these posteriors are not indepedent). The point-estimates of your parameters won't be biased, at least in the first order (i.e. their values won't tend to be systematically over- or underestimated).

  • $\begingroup$ Thank you Ruben. Your answer clarifies my question about the highly correlated samples I observed. $\endgroup$ – Nayomi Jan 24 '17 at 2:14

The serial correlation does not affect the mean values of the parameters, but the uncertainty on the mean value. You have to estimate the statistical efficiency (related to the serial correlation length) and use it to correct the number of samples. For a parameter $p$, the uncertainty on the mean becomes $\sigma/\sqrt{\rho*N}$ where $\sigma$ is the standard deviation of $p$, $0\le\rho\le1$ is the statistical efficiency and $N$ the sample size.

For instance, the function coda::effectiveSize() in R gives you directly the product $\rho*N$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.