1
$\begingroup$

I have two studies, let's call them A and B. Both involve fitting a linear mixed model on what is essentially the same type of data; the two studies differ in only small ways. In A I found a significant (p < .001) slope for my condition as a fixed effect. The theory I'm working with predicts that there should be no effect of condition in B (or almost no effect), so my hope is that I can demonstrate that the data from study B is more consistent with a slope of zero than with the data from study A.

(I'd be comfortable running this as just a linear regression if necessary, as the random effects in the model do very little.)

My understanding is that a Bayesian approach is the best way to provide 'evidence for the null', so I have been trying to find a way to do the analysis described above. My understanding is also that a Bayes Factor will be the measure I want, since it essentially describes the relative strength of the evidence in the data for two priors (?). But finding a way to actually conduct this analysis has been challenging.

I've investigated several R packages that seem like they could do this job, including blme, BayesRS, and BayesFactor. This doesn't seem to be the normal use case for anything I've investigated, and I'm still not even sure how to define a prior based on my data from A. Most of what I've seen has just demonstrated how to compare Bayesian regression models to one another.

Can anyone point me in the direction of the next steps for this? At this point I'm not sure what I'm missing or where else to search.

$\endgroup$

2 Answers 2

1
$\begingroup$

Welcome to CV.

Questions about code are off-topic here, but most of your question seems to be about statistics, so that's OK.

If you are trying to show evidence in favor of "no effect" the usual method is equivalence testing. One method of equivalence testing is TOST (two one sided t-tests). Searching for this term should help you find things.

An alternative that may suit you is to combine the two samples and then include the sample (A vs B) as an independent variable. You could then look into mediation models.

I am not sure why you think a Bayesian approach would be good here, but I am not at all expert on Bayesian methods.

$\endgroup$
2
  • $\begingroup$ The Bayesian approach is preferable since, unlike the frequentest approach, Bayesian statistics can be used to quantify evidence in favor of the null hypothesis. $\endgroup$ Commented Sep 19, 2018 at 16:57
  • $\begingroup$ @williamstome that is just one argument. There are so many other considerations. The suitability of a Bayesian approach depends a lot on whether one has suitable prior information and whether one is looking for an update or not. Besides that, the en.wikipedia.org/wiki/Equivalence_test is quantative (albeit just a bit more liberal/simplistic in prior assumptions and uses only a single cut-off level, equivalence bound, instead of an entire prior believe about the distribution of effects), and is very usefull in many situations. $\endgroup$ Commented Sep 19, 2018 at 17:27
0
$\begingroup$

Have you looked at JASP? It's very easy to use -- and the community is VERY helpful. I'd post your question on the JASP forums, I'm sure you'll get a helpful response.

I'm pretty new to Bayesian stats as well, but I assume what you want to do is analyze each experiment separately, testing simply to see if there is an effect of condition. This will yield, for each experiment, a Bf quantifying the ratio of evidence for this hypothesis to the evidence against this hypothesis. If you take the inverse of this Bf, you will get the evidence in favor of there being no difference in each experiment. I believe you can then look at the ratio between those Bfs to get what you're looking for, but I'm not confident in that last step.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.