Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

I'm analysing reaction time data using mixed-effect modelling in R. Data comes from 2 types of participant groups: native speakers and non-native speakers. For the non-natives, I have proficiency scores (estimating their mastery of English). The proficiency of native speakers is irrelevant, and coded as NA. Does this mean that lmer will consider proficiency as a factor only for non-native speakers?

'data.frame':   8373 obs. of  17 variables:
$Subject       : Factor w/ 21 levels 
$L1            : Factor w/ 3 levels "English","German",..: 
$Proficiency   : Factor w/ 12 levels:"0","0.6","0.61",..: 8 8 8 8 8 8 8 8 8 8 ...  
$Target        : Factor w/ 243 levels 
$Relation      : Factor w/ 4 levels 
$Word.Order    : Factor w/ 2 levels "HeadMod*","ModHead"
$Priming       : Factor w/ 2 levels "PrHead","PrMod"
$Trial         : Factor w/ 481 levels 
$Target.RTinv  : num

I'm concerned that when I add Proficiency to my model, the AIC and BIC become negative. Is this something to be concerned about?

Models:
dat.lmer5: -1000 * Target.RTinv ~ (1 | Subject) + (1 | Target) + L1 + Word.Order + Priming
dat.lmer8: -1000 * Target.RTinv ~ (1 | Subject) + (1 | Target) + L1 + Word.Order + Priming + Proficiency
          Df     AIC     BIC  logLik  Chisq Chi Df Pr(>Chisq)
dat.lmer5  8 1859.68 1915.92 -921.84
dat.lmer8 17 -438.62 -329.59  236.31 2316.3      9  < 2.2e-16 ***
share|improve this question
Can you post a str of your data and the call to lmer? However, from your description I have a wild guesses: Yes, lmer only uses proficiency scores when available. – Henrik Apr 24 '12 at 10:37
dat.lmer8 = lmer(-1000 * Target.RTinv ~ (1|Subject) + (1|Target) + L1 + Word.Order + Priming + Proficiency, dat, REML=TRUE) – cecile Apr 24 '12 at 11:11
My guess (I am almost sure, but Henrik's comment is making me a little uncertain) is that if you include proficiency as a factor, all observations with proficiency = NA are excluded, so you would not actually have any native speakers left! – Erik Apr 24 '12 at 14:39
Why is proficiency a factor? It looks continuous from the little information given. (0, 0.6, 0.61, ...) – Aaron Apr 24 '12 at 14:42

1 Answer

up vote 2 down vote accepted

It only uses data from complete cases, so the ones with NA are not being included in the second model at all. This makes the AIC/BIC incomparable, as it only makes sense to use AIC/BIC to compare models based on the same data.

Look at the output from summary to confirm, and to see how many observations were included in each model.

The negative isn't a problem, though; the actual value has little interpretable meaning; what matters is how different the values are. Smaller is always better; so if they're both negative, the more negative value is better.

share|improve this answer
Thanks a lot for the very clear explanation. How do you suggest I take Proficiency into account if I want to keep all my participants in, even those without a proficiency score? (And I guess you're right, it should be treated as continuous, so that was a newbie mistake to start with...) – cecile Apr 24 '12 at 15:16
Just give all native speakers the same proficiency value; they'll get a different offset because of the native/non-native term. (0 is probably preferred for interpretation of the coefficients, but technically, any value will result in the same fit.) – Aaron Apr 24 '12 at 15:27
Perfect. Works fine now, and (FYI) the AIC and BIC now remain positive. Thanks! – cecile Apr 24 '12 at 21:27
Please allow me a follow-up: How can I ascertain the effect of proficiency for the non-natives? I've tried modelling it as an interaction between language group and proficiency, but I get an error. dat.lmer14 = lmer(-1000 * Target.RTinv ~ (1|Subject) + (1|Target) + Proficiency*L1 + (Word.Order + L1 + Priming)^3, dat, REML=TRUE) Error in mer_finalize(ans) : Downdated X'X is not positive definite, 8. – cecile Apr 25 '12 at 11:41
The effect of proficiency is only for the non-natives because they're the only ones with proficiency values; you don't need the interaction. – Aaron Apr 25 '12 at 13:27
show 7 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.