7
$\begingroup$

I heard somewhere, that I can directly test (or gather support for) a null-hypothesis using the Bayes-Factor. In my specific experiment, I hypothesize that an experimental manipulation does not have an impact on some variable but does selectively impact another one. Somehow, simply showing that a t-test gives non-significant results does not seem appropriate (because this can only "not-reject" the H0).

My specific problem is as follows:

I have two experimental conditions, say medication and control, and I would like to show that estimated parameter-values are the same in both conditions (I measure some data $X$ and have a model with parameters $\alpha,\beta$ giving the likelihood $P(X|\alpha,\beta)$. I measured $N$ subjects in both the control and medication condition (repeated measures).

My first approach would be to set up a hierarchical model where $\alpha,\beta$ are distributed for each individual according to some group-level distribution (since both parameters have to be positive and are more likely to be small, I could use an exponential). The group-level distribution would have a uniform prior in a feasible range.

I would use this model to sample (MCMC) from the posterior and I think I would be interested in comparing the group-level estimates. However it is unclear to me, how to integrate the two competing hypotheses (H0 being that $\alpha_{med}=\alpha_{cntrl}$ and H1 being that $\alpha_{med}\ne\alpha_{ctrl}$ and the same for $\beta$).

So my question is: How can I go from this model to the Bayes-factor?

Concretely, I am running my MCMC in the pymc package so any help with concrete code would help a lot!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

You could try the approach recommended by Steve Goodman and calculate the minimum bayes factor: Toward Evidence Based Medical Statistics 2: The Bayes Factor

To get this from mcmc results, you can subtract the estimate for the group level parameters for each step to get a posterior distribution of the difference as was done by John Kruschke in this paper: Bayesian Estimation Supersedes the t Test

He does not calculate a bayes factor there and recommends against it (see appendix D). Instead he designates a region of practical equivalence around the null hypothesis (zero) and see if your credible interval overlaps.

To get the minimum bayes factor I believe what you can do is then divide the probability at the mode of your estimate of the difference between means by the probability at zero. I have not seen this done anywhere but it makes sense to me. Hopefully someone else can comment on that.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for these two excellent references, I will read up on it and continue the discussion later... $\endgroup$
    – thias
    Commented Nov 20, 2013 at 8:27
  • $\begingroup$ "he designates a region of practical equivalence around the null hypothesis (zero) and see if your credible interval overlaps." --> This sounds awfully like confidence intervals inference as advocated by the likes of George Cumming and other estimation statistics proponents, with potentially the limitations of overlap inference as highlighted by Stephen Senn. $\endgroup$
    – gaborous
    Commented Sep 4, 2019 at 16:13
3
$\begingroup$

You can use the BayesFactor package in R to easily compute Bayesian t tests. See the examples here: http://bayesfactorpcl.r-forge.r-project.org/#twosample for details. The web calculator at http://pcl.missouri.edu/bayesfactor uses the same models (see the Rouder et al 2009 reference on the web calculator page). Note that the Kruschke reference given above does not actually allow you to test a null hypothesis.

Improve this answer
$\endgroup$
4
  • $\begingroup$ thanks for the link. As far as I can tell, the bayesfactor package implements a bayesian t-test which takes two samples as input. While I could do this on the subject-level, I would also be interested in directly comparing the group-level estimates from the hierarchical model. Is this possible using this package, too? $\endgroup$
    – thias
    Commented Jan 28, 2014 at 8:36
  • $\begingroup$ The BayesFactor package implements Bayesian Linear Mixed Effect models. If you're happy living with the assumptions of that class of models (linear effects, independent normal error terms) then you can use the package. I can't say much more without knowing your application. $\endgroup$
    – richarddmorey
    Commented Jan 28, 2014 at 20:15
  • $\begingroup$ what I meant in my previous comment, that I would like to compare the group-level posterior distributions from a hierarchical bayesian model. Say that my model is built such that the $i$'th subjects $\alpha_{med,i}$ and $\alpha_{ctrl,i}$ parameters are distributed according to a normal distribution with mean $\mu_{\alpha_{med}}$ vs. $\mu_{\alpha_{ctrl}}$, how can I compare the posterior distributions for $\mu_{\alpha_{med}}$ vs $\mu_{\alpha_{ctrl}}$? These posterior are defined by their MCMC samples, so just comparing samples with t-test does not seem to be reasonable? $\endgroup$
    – thias
    Commented Jan 29, 2014 at 13:02
  • $\begingroup$ So, if I understand, you could define $\alpha_{ci} = \mu + \gamma_c + \delta_i + \epsilon_{ci}$ where $c$ indexes conditions, $i$ indexes participants, and $\mu$, $\gamma$, and $\delta$ are the effects of the grand intercept, condition, and participant, respectively? If that's the case, you can use the BayesFactor package. You can email me for further help if you like. You'll use generalTestBF(). $\endgroup$
    – richarddmorey
    Commented Feb 3, 2014 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.