We were asked by a reviewer to provide p-values as to better understand the model estimates in our bayesian multilevel model. The model is a typical model of multiple observations per participant in an experiment. We estimated the model with Stan, so we can easily compute additional posterior statistics. Currently, we are reporting (visually and in tables) the mean estimate and the 0.025 and 0.975 quantiles.
My response so far would include:
- P-values are inconsistent with bayesian models, i.e. $P(X|\theta) \neq P(\theta|X).$
- Based on the posterior, we can calculate the probability of parameters being larger (smaller) than 0. This looks a bit like a traditional p-value.
My question is whether this is a response that can satisfy a reviewer or will it only cause more confusion?
Update 10-Oct: We rewrote the paper with the advice in the answer in mind. The paper is accepted so I will reiterate my earlier comment that this was really helpful advice!