Providing CIs for comparing interaction point estimates across datasets?

Question

I have more of a brainstorming type of question for statisticians and I would like to know what your thoughts are. I'll describe my data sets, so you can judge what type of statistical problem they involve.

I have two different language experiments: one was done with Spanish speakers and one was done with English speakers. Both experiments have exactly the same design: a 2 by 2 within subjects design, crossing two factors, "grammaticality" (grammatical/ungrammatical word) and "number" (singular/plural word). I am interested in the interaction between these two factors. When I run a my statistical model (a factorial anova using lmer in R) I get a significant interaction in both cases: beta / the interaction coefficient is significantly different from 0, with p <0.05

My goal is to compare the value of beta between the two data sets (since I have two experiments, I have one beta value for the Spanish experiment and one beta value for the English experiment). In order to do that, I was thinking of providing the beta value for each experiment and getting a confidence interval for each beta, with the goal of showing that the two values (and their CIs) are not that different.

I am trying to figure out which is the most appropriate way of calculating the 95% CIs. I thought of doing bootstrapping as a way of getting them. So I guess my questions are:

1. Is this way (providing the betas for English and Spanish and the 95% bootstrapped CIs?) a clear way of comparing across experiments? (that is, are there other more appropriate statistical ways of making the comparison across languages?)

2. Is bootstrapping an adequate way of getting the CIs? And if not, how would you get them?

Let me know if this was unclear or if you need me to expand and thanks!

Answer 1

Your software should provide a standard error for your beta coeffcient for the interaction. Baayen (2008) recommends the following for calculating CI's of the coefficients and it uses bootstrapping.

# assuming m is your lmer model
mc <- MCMCSamp(m, 1000)
HPDinterval(mc)$fixef

You seem to be unclear what the beta value actually represents. I could be wrong but for other readers as well, in a 2x2 it should equal (A1 - A2) - (B1 - B2). And because it's an effect the CI can be used in the traditional way across experiments.

You might have been thinking of confidence intervals around predicted values when you have repeated measures (something like the old Loftus & Masson recommendation or "narrow inference" CI, Blouin & Riopelle; 2005). Those often can't be compared across experiments. In repeated measures and multi-level modelling a CI around the predicted condition value is meaningful across experiments or within but not both.