I'm reading a paper where the authors have something like the following as one step in their MCMC:

$$ \begin{align} y&=\rho_1 x_1+\rho_2 x_2+\epsilon\\ z&=\beta\tilde{y}+u \end{align} $$ where $\tilde{y}=\frac{1}{1-\rho_1-\rho_2}(y+\rho_2 x_1)$, $\epsilon$ and $u$ are independent normals and $\beta$ and $\rho$ are parameters of interest. What is bugging me is that to sample from the (conditional) posterior of $\rho$, only the first equation is used, which makes the conditional posterior normal under a normal prior and sampling is easy. But $\rho$ shows up also in the second equation through $\tilde{y}$, meaning that they essentially assume $$ p(\rho|\beta, y, z)=p(\rho|y) $$ but I don't see how that is justified here.

My question is: does neglecting the presence of $\rho$ in the second equation create any problems? In some sense, it seems to me like this is some kind of partial posterior, which may not be "efficient" but not necessarily "incorrect" (I know I'm vague here).

Edit: The paper is here: http://users.ugent.be/~geeverae/index_files/US%20TVP%20model.pdf and the related parts are equations (4), (14) and section Block 1(d) in the Appendix.

  • $\begingroup$ I think having $\beta$ in the second equation multiplying $1/(1-\rho)$ means that $\rho$ doesn't matter, as what really matters is the ratio $\beta/(1-\rho)$, and no matter what $\rho$ is, there's a $\beta$ that will give you whatever value of the ratio you want. A shorter way of putting it is that $\beta$ and $\rho$ are not both identifiable in the second equation, so fixing $\rho$ at whatever the sampled value from the first equation is at least allows you to sample $\beta | \rho$ for the second equation. $\endgroup$ – jbowman Oct 24 '18 at 19:52
  • $\begingroup$ @jbowman that is a good point. I think maybe I simplified the problem too much so that it became trivial! I have updated the post so that it more resembles my real problem; there’s in fact two parameters and I’m not sure that idea goes through now too? $\endgroup$ – hejseb Oct 24 '18 at 20:06
  • 1
    $\begingroup$ If the equations for the model are as given, then indeed the conditional of $Z$ given $Y$ is also indexed by $\rho$ and not accounting for this in the simulation of $\rho$ is incorrect. $\endgroup$ – Xi'an Oct 25 '18 at 3:02

Here is a small numerical example of the issue.

# Generate data
r1 <- 0.5
r2 <- 0.3
b1 <- 0.5
x_1 <- rnorm(100)
x_2 <- rnorm(100)
y <- r1*x_1 + r2*x_2 + rnorm(100)
z <- b1/(1-r1-r2)*(y+r2*x_1) + rnorm(100)

# Log likelihood function
log_like <- function(r, y, z, x_1, x_2, b1) {
  r1 <- r[1]
  r2 <- r[2]
  sum(dnorm(y-r1*x_1-r2*x_2, log = TRUE) + dnorm(z - b1/(1-r1-r2)*(y+r2*x_1), log = TRUE))

# Analytical osterior using only first equation
X <- cbind(x_1, x_2)
r_mean <- chol2inv(chol(crossprod(X))) %*% crossprod(X, y)
r_cov  <- chol2inv(chol(crossprod(X)))

# MCMC settings 
n_reps <- 100000
rho_draws <- matrix(0, n_reps, 2)
step_size <- 0.02
r_curr <- c(r1, r2)
accept <- numeric(n_reps)

# Metropolis-Hastings using random walk proposal
for (i in 1:n_reps) {
  r_prop <- rnorm(2, r_curr, step_size)
  alpha <- exp(log_like(r_prop, y, z, x_1, x_2, b1) - log_like(r_curr, y, z, x_1, x_2, b1))
  u <- runif(1)
  if (u < alpha) {
    r_curr <- r_prop
    accept[i] <- 1
  rho_draws[i, ] <- r_curr
r_draws_partial <- MASS::mvrnorm(n_reps, r_mean, r_cov)

# Plot the result
tibble(values = c(c(rho_draws), c(r_draws_partial)), 
       par = rep(c(1, 2, 1, 2), each = n_reps),
       posterior = rep(c("full", "partial"), each = 2*n_reps)) %>%
  ggplot(aes(x=values)) +
  geom_density(aes(fill = posterior), alpha = 0.5) +
  geom_vline(data = tibble(true = c(r1, r2), par = 1:2), 
             mapping = aes(xintercept = true),
             size = 1) +
  facet_wrap(~par, scales = "free")

enter image description here

The vertical lines are the true values for $\rho_1$ and $\rho_2$, the blue density curve is based on the first equation only (using the analytical "partial" posterior) and the red is based on Metropolis-Hastings using both equations. Thus, using the approach suggested in the paper is probably not very wise.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.