4
$\begingroup$

I have more of a brainstorming type of question for statisticians and I would like to know what your thoughts are. I'll describe my data sets, so you can judge what type of statistical problem they involve.

I have two different language experiments: one was done with Spanish speakers and one was done with English speakers. Both experiments have exactly the same design: a 2 by 2 within subjects design, crossing two factors, "grammaticality" (grammatical/ungrammatical word) and "number" (singular/plural word). I am interested in the interaction between these two factors. When I run a my statistical model (a factorial anova using lmer in R) I get a significant interaction in both cases: beta / the interaction coefficient is significantly different from 0, with p <0.05

My goal is to compare the value of beta between the two data sets (since I have two experiments, I have one beta value for the Spanish experiment and one beta value for the English experiment). In order to do that, I was thinking of providing the beta value for each experiment and getting a confidence interval for each beta, with the goal of showing that the two values (and their CIs) are not that different.

I am trying to figure out which is the most appropriate way of calculating the 95% CIs. I thought of doing bootstrapping as a way of getting them. So I guess my questions are:

  1. Is this way (providing the betas for English and Spanish and the 95% bootstrapped CIs?) a clear way of comparing across experiments? (that is, are there other more appropriate statistical ways of making the comparison across languages?)

  2. Is bootstrapping an adequate way of getting the CIs? And if not, how would you get them?

Let me know if this was unclear or if you need me to expand and thanks!

$\endgroup$
5
$\begingroup$

Your software should provide a standard error for your beta coeffcient for the interaction. Baayen (2008) recommends the following for calculating CI's of the coefficients and it uses bootstrapping.

# assuming m is your lmer model
mc <- MCMCSamp(m, 1000)
HPDinterval(mc)$fixef

You seem to be unclear what the beta value actually represents. I could be wrong but for other readers as well, in a 2x2 it should equal (A1 - A2) - (B1 - B2). And because it's an effect the CI can be used in the traditional way across experiments.

You might have been thinking of confidence intervals around predicted values when you have repeated measures (something like the old Loftus & Masson recommendation or "narrow inference" CI, Blouin & Riopelle; 2005). Those often can't be compared across experiments. In repeated measures and multi-level modelling a CI around the predicted condition value is meaningful across experiments or within but not both.

$\endgroup$
  • $\begingroup$ Thanks @John! I know what you mean and I have been calculating CIs in that way in the past. I was just wondering if that was the most appropriate way to do it when your goal is to compare the beta coefficients across two different samples (with different subjects and items). My goal is to be able to conclude how close to each other the betas actually are, and I wasn't sure whether the CIs generated in Baayen's way would be an appropriate way. $\endgroup$ – Sol Jul 11 '13 at 12:07
  • $\begingroup$ Thanks again! This is very useful. One last question. What would you do if you wanted to run a statistical test comparing the values of the two betas? What statistical test would you run? $\endgroup$ – Sol Jul 12 '13 at 12:49
  • 1
    $\begingroup$ I'd look at a document on how to read and interpret confidence intervals. I generally am not much interested in tests per se. Check out Cumming and Finch 2005 as a good primer. $\endgroup$ – John Jul 12 '13 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.