10
$\begingroup$

Suppose that, in a study of 15 subjects, the response variable (res) is modeled with two explanatory variables, one (level) is categorical with 5 levels and the other (response time: RT) is continuous. With lmer in the lme4 package of R, I have:

fm1 <- lmer(res ~ level * RT + (level-1 | subject), data=mydata)
anova(fm1)

             Df  Sum Sq Mean Sq  F value
level        4  3974.9   993.7   9.2181
RT           1  1953.5  1953.5  18.1209
level:RT     4  5191.4  1297.9  12.0393

If I change the order of the two variables, I get slightly different results for the main effects:

fm2 <- lmer(res ~ RT * level + (level-1 | subject), data=mydata)
anova(fm2)

             Df  Sum Sq Mean Sq  F value
RT           1  1671.8  1671.8  15.5077
level        4  4256.7  1064.2   9.8715
RT:level     4  5191.4  1297.9  12.0393

Does such a difference come from the sequential (instead of marginal) approach in lme4 in accounting for data variability? In this case, the variable order change does not lead to a big difference, but previously I've seen dramatic differences. What does such a big difference mean? Does it mean that the model needs more tuning until big difference disappears?

My second question is that, if I want to know which variable among the two (RT and level) accounts for more data variability, what would be a reasonable approach? Based on the relative magnitude of Sum Sq (or Mean Sq) of the two variables? Any statistical testing method to compare variability among explanatory variables?

Improve this question
$\endgroup$
1
  • $\begingroup$ If you can't post data can you at least put up the model output rather than just the ANOVA? $\endgroup$
    – John
    Commented Aug 19, 2011 at 19:57

1 Answer 1

Reset to default
7
$\begingroup$

I will try to answer your questions one-by-one:

Does such a difference come from the sequential (instead of marginal) approach in lme4 in accounting for data variability?

Correct. As you can see, only for the interaction are the results the same. The interaction is entered last into the model in both cases, so the results for that term are the same. However, if you enter "level" first and then "RT", the results for "RT" tell you whether "RT" is significant after "level" is already in the model (and vice-versa). These results are order-dependent.

What does such a big difference mean?

Suppose both variables by themselves are strongly related to the response variable, but they are also strongly correlated. In that case, there may not be a whole lot of variability in the response variable left to account for by the variable that is entered second into the model. Therefore, you will tend to see more dramatic differences when the explanatory variables are correlated.

Does it mean that the model needs more tuning until big difference disappears?

I am not sure what you mean by "tuning". The phenomenon you are observing is not a problem per se, although it does complicate the interpretation of the results (see below).

Maybe one way of "tuning" is this. If the explanatory variables are highly correlated, then they may essentially be measuring the same thing. In that case, one can "tune" the model by either removing one of the variables or combining them into a single variable.

My second question is that, if I want to know which variable among the two (RT and level) accounts for more data variability, what would be a reasonable approach? Based on the relative magnitude of Sum Sq (or Mean Sq) of the two variables? Any statistical testing method to compare variability among explanatory variables?

When the explanatory variables are correlated, then it is rather difficult to determine their relative importance. This issue comes up quite frequently in the multiple regression context and dozens of articles have been written on this topic and lots of methods for accomplishing this goal have been suggested. There certainly is no consensus on the most appropriate way and some people may even suggest that there is no adequate way of doing that.

The sums of squares are not going to help you, because they are not based on the same number of degrees of freedom. The mean squares essentially correct for that, but if you use the mean squares, then this is nothing else than using the corresponding F-values (or p-values) to determine the relative importance. I think most people would not consider that an appropriate way of determining the relative importance.

Unfortunately, I do not have an easy solution. Instead, I can suggest a website to you, from the author of the relaimpo package. I don't think the package will help you when fitting mixed-effects models, but there are lots of references to papers on the issue you are dealing with.

http://prof.beuth-hochschule.de/groemping/relaimpo/

You may also want to look into the AICcmodavg package:

http://cran.r-project.org/web/packages/AICcmodavg/index.html

Improve this answer
$\endgroup$
3
  • $\begingroup$ Really appreciate the clarifications and help! Regarding the relative importance of each variable in a mixed-effect model, is it reasonable to consider the AIC/BIC difference provided by lme()? For example, with my data define three models: res ~ RT, res ~ level, res ~ RT * level. Then use anova() to assess relative importance based on their AIC/BIC? $\endgroup$
    – bluepole
    Commented Aug 22, 2011 at 17:52
  • $\begingroup$ AIC/BIC values are often used for model selection. If that would be your goal, then yes, this is one approach. If your goal is to determine the relative importance of the variables, then you could use Akaike weights. A very nice and clear presentation on that idea can be found here: bit.ly/pDQ7eq $\endgroup$
    – Wolfgang
    Commented Aug 23, 2011 at 8:09
  • $\begingroup$ Thanks a lot for all the help. I really appreciate it... $\endgroup$
    – bluepole
    Commented Aug 24, 2011 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.