Skip to main content
Tweeted twitter.com/#!/StackStats/status/425596080106258432
added 321 characters in body
Source Link
Wayne
  • 21.6k
  • 4
  • 58
  • 111

Say I have run a Hierarchical Bayesian model in STAN (or JAGS or BUGS) and I have the posterior samples of two slope parameters that I want to compare: $\beta_1$ and $\beta_2$. The model appears to have converged properly.

So I look at the density of the difference of the MCMC samples of the posteriors of the two slopes ($\beta_1-\beta_2$). The 95% HPD interval of this difference includes zero, and from what I've read this means something similar to a Frequentist failure-to-reject: zero is credible and hence we cannot say that $\beta_1 \ne \beta_2$.

Yet, I can see that 80% of the differences are greater than zero. Which tempts me to say that, given my assumptions and model the odds are 4-1 that $\beta_1>\beta_2$. But I don't think I've ever read anything like that.

This is so simple that it would be used everywhere if it were correct. So it must not be correct: what am I missing?

EDITEDIT2: The more I think about itOK, the more it seems correctI found an answer to my question. The issue would then be are 4-1 odds very good? The canonical 95% CI is 19-1 oddsprevious paragraph describes a decision rule, so in light of that 4-1which is not very impressive and hencedistinct from the model, MCMC, etc. This particular decision rule has two problems:

  1. It doesn't take into account the density of the differences. That is, the 80% of the differences that are greater than zero might all be very close to zero, or might be quite distant from zero. This should make a difference, but the simplistic decision rule doesn't differentiate between these conditions.

  2. The 4-1 odds are not all that strong to begin with, considering that the conventional 95% CI is essentially 19-1 odds.

I'm not worth mentioningsure how to make a better decision rule, though. Is this it?

Say I have run a Hierarchical Bayesian model in STAN (or JAGS or BUGS) and I have the posterior samples of two slope parameters that I want to compare: $\beta_1$ and $\beta_2$. The model appears to have converged properly.

So I look at the density of the difference of the MCMC samples of the posteriors of the two slopes ($\beta_1-\beta_2$). The 95% HPD interval of this difference includes zero, and from what I've read this means something similar to a Frequentist failure-to-reject: zero is credible and hence we cannot say that $\beta_1 \ne \beta_2$.

Yet, I can see that 80% of the differences are greater than zero. Which tempts me to say that, given my assumptions and model the odds are 4-1 that $\beta_1>\beta_2$. But I don't think I've ever read anything like that.

This is so simple that it would be used everywhere if it were correct. So it must not be correct: what am I missing?

EDIT: The more I think about it, the more it seems correct. The issue would then be are 4-1 odds very good? The canonical 95% CI is 19-1 odds, so in light of that 4-1 is not very impressive and hence not worth mentioning. Is this it?

Say I have run a Hierarchical Bayesian model in STAN (or JAGS or BUGS) and I have the posterior samples of two slope parameters that I want to compare: $\beta_1$ and $\beta_2$. The model appears to have converged properly.

So I look at the density of the difference of the MCMC samples of the posteriors of the two slopes ($\beta_1-\beta_2$). The 95% HPD interval of this difference includes zero, and from what I've read this means something similar to a Frequentist failure-to-reject: zero is credible and hence we cannot say that $\beta_1 \ne \beta_2$.

Yet, I can see that 80% of the differences are greater than zero. Which tempts me to say that, given my assumptions and model the odds are 4-1 that $\beta_1>\beta_2$. But I don't think I've ever read anything like that.

EDIT2: OK, I found an answer to my question. The previous paragraph describes a decision rule, which is distinct from the model, MCMC, etc. This particular decision rule has two problems:

  1. It doesn't take into account the density of the differences. That is, the 80% of the differences that are greater than zero might all be very close to zero, or might be quite distant from zero. This should make a difference, but the simplistic decision rule doesn't differentiate between these conditions.

  2. The 4-1 odds are not all that strong to begin with, considering that the conventional 95% CI is essentially 19-1 odds.

I'm not sure how to make a better decision rule, though.

added 237 characters in body
Source Link
Wayne
  • 21.6k
  • 4
  • 58
  • 111

Say I have run a Hierarchical Bayesian model in STAN (or JAGS or BUGS) and I have the posterior samples of two slope parameters that I want to compare: $\beta_1$ and $\beta_2$. The model appears to have converged properly.

So I look at the density of the difference of the MCMC samples of the posteriors of the two slopes ($\beta_1-\beta_2$). The 95% HPD interval of this difference includes zero, and from what I've read this means something similar to a Frequentist failure-to-reject: zero is credible and hence we cannot say that $\beta_1 \ne \beta_2$.

Yet, I can see that 80% of the differences are greater than zero. Which tempts me to say that, given my assumptions and model the odds are 4-1 that $\beta_1>\beta_2$. But I don't think I've ever read anything like that.

This is so simple that it would be used everywhere if it were correct. So it must not be correct: what am I missing?

EDIT: The more I think about it, the more it seems correct. The issue would then be are 4-1 odds very good? The canonical 95% CI is 19-1 odds, so in light of that 4-1 is not very impressive and hence not worth mentioning. Is this it?

Say I have run a Hierarchical Bayesian model in STAN (or JAGS or BUGS) and I have the posterior samples of two slope parameters that I want to compare: $\beta_1$ and $\beta_2$. The model appears to have converged properly.

So I look at the density of the difference of the MCMC samples of the posteriors of the two slopes ($\beta_1-\beta_2$). The 95% HPD interval of this difference includes zero, and from what I've read this means something similar to a Frequentist failure-to-reject: zero is credible and hence we cannot say that $\beta_1 \ne \beta_2$.

Yet, I can see that 80% of the differences are greater than zero. Which tempts me to say that, given my assumptions and model the odds are 4-1 that $\beta_1>\beta_2$. But I don't think I've ever read anything like that.

This is so simple that it would be used everywhere if it were correct. So it must not be correct: what am I missing?

Say I have run a Hierarchical Bayesian model in STAN (or JAGS or BUGS) and I have the posterior samples of two slope parameters that I want to compare: $\beta_1$ and $\beta_2$. The model appears to have converged properly.

So I look at the density of the difference of the MCMC samples of the posteriors of the two slopes ($\beta_1-\beta_2$). The 95% HPD interval of this difference includes zero, and from what I've read this means something similar to a Frequentist failure-to-reject: zero is credible and hence we cannot say that $\beta_1 \ne \beta_2$.

Yet, I can see that 80% of the differences are greater than zero. Which tempts me to say that, given my assumptions and model the odds are 4-1 that $\beta_1>\beta_2$. But I don't think I've ever read anything like that.

This is so simple that it would be used everywhere if it were correct. So it must not be correct: what am I missing?

EDIT: The more I think about it, the more it seems correct. The issue would then be are 4-1 odds very good? The canonical 95% CI is 19-1 odds, so in light of that 4-1 is not very impressive and hence not worth mentioning. Is this it?

Source Link
Wayne
  • 21.6k
  • 4
  • 58
  • 111

MCMC direct comparison of difference of two parameters

Say I have run a Hierarchical Bayesian model in STAN (or JAGS or BUGS) and I have the posterior samples of two slope parameters that I want to compare: $\beta_1$ and $\beta_2$. The model appears to have converged properly.

So I look at the density of the difference of the MCMC samples of the posteriors of the two slopes ($\beta_1-\beta_2$). The 95% HPD interval of this difference includes zero, and from what I've read this means something similar to a Frequentist failure-to-reject: zero is credible and hence we cannot say that $\beta_1 \ne \beta_2$.

Yet, I can see that 80% of the differences are greater than zero. Which tempts me to say that, given my assumptions and model the odds are 4-1 that $\beta_1>\beta_2$. But I don't think I've ever read anything like that.

This is so simple that it would be used everywhere if it were correct. So it must not be correct: what am I missing?