Specification and Interpretation of Repeated Measures Binomial model in BRMS

Question

I have two questions regarding specifying and interpreting Repeated Measures Binomial Models in BRMS

We have a set of data in the following format:

+----+------+----------+
| ID | Cond | Response |
+----+------+----------+
|  1 |    0 |        0 |
|  1 |    0 |        0 |
|  1 |    0 |        1 |
|  2 |    1 |        1 |
|  2 |    1 |        1 |
|  2 |    1 |        0 |
|  3 |    0 |        0 |
|  3 |    0 |        1 |
|  3 |    0 |        0 |
+----+------+----------+

That is, three yes/no measures of Response are taken for each participant, each of whom is assigned to one of two conditions.

We would like to determine whether Condition has an effect on Response, using BRMS.

From what I can tell, the way to specify this model is:

Response | trials(1) ~ Condition + (1 |ID)

Specified more fully as:

fit <- brm(Response | trials(1) ~ Condition + (1 | ID), data=mydata, family=binomial, prior = prior(normal(0, 10), class = b))

Question 1: Is this the correct model specification?

This gives us the following model output:

Group-Level Effects: 
~ID (Number of levels: 30) 
              Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
sd(Intercept)     8.71      3.56     3.91    17.65       1264 1.00

Population-Level Effects: 
          Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
Intercept     8.72      3.67     3.29    17.60        886 1.01
CondB        -7.83      4.38   -17.66    -0.78        882 1.00

We're now trying to interpret these results.

Question 2: Is the following the correct way to interpret this model?

Response is 7.83 times more likely [0.78,17.66] to be 1 in Condition 0 than in Condition 1

Bonus question: We're not entirely clear on how best to select and specify priors. We have some prior evidence suggesting that when some other factor was changed (not the same thing as our Cond), Response changed from a rate of 1:4 to a rate of 3:1; a 12x change in odds. But we're unclear how to act on this information and use it to select a reasonable prior. Based on this blog post, we tried setting the Prior to normal(0,1); normal(0,10); and normal(0,50). Unfortunately, this produced wildly different results:

normal(0,1) produces an intercept of 4.83 and a CondB coefficient of -0.61.

normal(0,10) produces an intercept of 8.72 and a CondB coefficient of -7.83.

normal(0,50) produces an intercept of 10.68 and a CondB coefficient of -10.51.

When we compare these to a null model

null_fit <- brm(Response | trials(1) ~ (1 | ID), data=mydata, family=binomial, save_all_pars=TRUE)

using bayes_factor(fit,null_fit), we get Bayes Factors of, respectively, 1.14, 4.26, 1.40.

This is quite distressing, as this suggests that our results are very sensitive to choice of prior (weak evidence in favor of an effect when we use the medium-weight prior... and a toss up when we use the strongly informative or weakly informative prior.)

We're not sure what to do now, since we're not sure why we would choose one prior over another, and don't want to do the Bayesian equivalent of p-hacking.

So, bonus question: Ah, Help, Oh No, What Do We Do.

Answer 1

Even though I am currently not familiar with brms, logically speaking you are fitting a mixed effects logistic regression. In this case the intercept is the log odds when Cond is zero, and the coefficient for Cond is the log odds ratio between Cond=1 and Cond=0. Note though that the interpretation of these coefficients is conditional on the random intercepts you have included; for more information on this check here.

In your output these coefficients are quite large in magnitude (i.e., to transform to the more easily interpretable scale of odds and odds ratio you need to exponentiate them). This in combination with the behavior you have observed from the different priors suggests that you potentially have a separation issue. In this case, a stronger prior for the fixed effects coefficients is preferable, e.g., the $\mathcal N(0, 1)$ you have tried or a Student’s t prior with mean zero, scale parameter one, and three degrees of freedom.