I have a simple experiment in which ~30 people responded to 100 words, half of which were of Type A and half of which were of Type B. I am using a mixed effects linear regression to predict reaction time differences between words of Type A and Type B. I am also using a ME logistic regression to predict accuracy differences. These models are being fit using Bayesian parameter estimation.

Ultimately I am interested in determining the probability that the null hypothesis is true given the data. I am going to do this using Bayes Factors to compare models with and without the parameter of interest (i.e., word type).

Based on the literature, the largest possible mean difference I would expect between Type A and Type B is 100 ms. The effect size that I'd expect to be most likely is around 30 ms. I should be clear that these are both what I would expect if there is an effect. I think the most likely expectation is to find no effect at all.

I am curious how to now turn this into a prior. The various advice I have found so far is to:

  • use a normal distribution centred at zero with a standard deviation equal half of the maximum expected effect size
  • use a normal distribution centred at zero with a standard deviation equal to the expected effect size
  • set a uniform distribution across all possible effect sizes

I'm wondering if there are other rules of thumb in setting up the prior based on a maximum possible effect size and/or expected effect size? Is it common practice to centre the prior at zero? Note that this is all for the purpose of computing a Bayes Factor for the effect of word type.

  • $\begingroup$ Why do you wish to use a Bayesian technique? What do you believe is the advantage? $\endgroup$ – Sextus Empiricus Apr 2 '18 at 20:15
  • $\begingroup$ To be able to comment on the probability of the null hypothesis being true given the data. $\endgroup$ – Dave Apr 2 '18 at 20:39
  • 1
    $\begingroup$ Why do you wish to make that comment? How are you gonna use it? $\endgroup$ – Sextus Empiricus Apr 2 '18 at 20:46
  • $\begingroup$ To publish a paper on the limits of a given phenomenon/effect. $\endgroup$ – Dave Apr 2 '18 at 20:51
  • 1
    $\begingroup$ As an experimental scientist, to publish a paper it is sufficient to provide the results along with a maximum likelihood estimate and confidence interval. There is no need to 'spoil' this with a (subjective) prior if the posterior that you obtain from it has no practical use. No data is gonna give (on it's own) the probability of the null hypothesis or the limits of an effect. The data is only working as making certain values more or less likely than before (and that, this pure likelihood, is what you wish to convey in your publication). $\endgroup$ – Sextus Empiricus Apr 2 '18 at 20:57

A good way to set priors is using prior predictive checks - basically you simulate new datasets from your model. If the simulated datasets are unrealistic in any way, it means your priors have problems.

Prior predictive checks have the advantage that they take complete structure of the model into account. If you however have only one effect in the model or have other reasons to believe you can set the prior independently for each parameter, and you have minimal bound $c_-$ and maximal bound $c_+$ than a good way is to choose a family and then find such parameters for the family that $P(\beta < c_-) = P(\beta > c_+) = \alpha$ where $\alpha$ is small, but not extremely small ($0.05$ tends to work nice).

Stan wiki has some more considerations: https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations

Note that it is IMHO wrong to a-priori exclude effects in the opposite direction than expected.

Also, Bayes factors tend to be very sensitive to the parts of priors that don't matter for normal inference (e.g. choosing $N(0,100)$ vs. $N(0,1000)$ when the ML estimate is 0.5).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.