I have recently faced a weird way of finding a significant variable.

Let the model be

$Cholesterol=Sex+FamilyHistory + Sex*FamilyHistory$

where the * denotes the interaction, sex and FamilyHistory are both two-level factors, and we assume that the study is nested inside cities. Further, assume that the fitted model is a linear mixed model with City in the random effect [random intercept].

Having the model above, we are interested to find out whether Sex has a significant effect on response.

What I normally do is to fit the model and look at the p-value of Sex coefficient in the marginal anova (simply using summary(nlme fit) in R). However, I have been told that I should make a null model like


Or maybe! ($Cholesterol=FamilyHistory + Sex*FamilyHistory$)

and compare it with the full model using ANOVA (simply anova(full.model, null.model) in R).

So which one do you recommend?

UPDATE: as @Stefan requested, I give an example in R using the built-in example on lme manual. Sorry that the example is in a different context.

Let the full model be [it is the same as fm2 in the lme example in R]

full <-
    distance ~ age + Sex,
    data = Orthodont,
    random = ~ 1,
    method = 'ML'

Now the goal of the analysis is to find whether Sex is significant or not (obviously on the response)!

What I do is to look at the p-value from the sex coefficient from the full model.

 r1 = summary(full)
                 Value  Std.Error DF   t-value      p-value
(Intercept) 17.7067130 0.83154591 80 21.293729 1.057125e-34
age          0.6601852 0.06209293 80 10.632212 5.740140e-17
SexFemale   -2.3210227 0.74306676 25 -3.123572 4.478461e-03

that suggests a significant (in $\alpha =10^{-2}$) deviation for females than males. However, I have been told that the true way of finding the Sex effect is to form a null model by excluding Sex and comparing the resulting model with the full one:

null <-
    distance ~ age , # Here Sex is removed
    data = Orthodont,
    random = ~ 1,
    method = 'ML'
anova(full, null)
   Model df      AIC      BIC    logLik   Test  L.Ratio p-value
null     1  4 451.3895 462.1181 -221.6948                        
full     2  5 444.8565 458.2671 -217.4282 1 vs 2 8.533057  0.0035

As you see the pvalue here ($0.0035$) is slightly smaller than the p-value from the full model ($4.478461e-03$). I understand that the statistics are different, in fact, anova uses the $\chi^2$ distribution and test of coefficients applies $T$ distribution. However, I need to know which one is the proper one!

  • 1
    $\begingroup$ Sorry I misunderstood your question. The example makes it clear now. I'll delete my answer as it doesn't make much sense. $\endgroup$ – Stefan Feb 10 '18 at 16:09
  • $\begingroup$ Since the t-test compares the individual model terms to the reference level (Intercept), I wouldn't use it either to figure out whether Sex is significant or not. Those p-values will change depending on what you define as your reference level. Since you are dealing with factors, I agree with @SalMangiafico that you could simply use car::Anova() to get a table that shows the significance of the individual factors, including Sex. Comparing models using anova(), tests (as far as I understand) whether the explanatory power of one model is higher than the other... $\endgroup$ – Stefan Feb 10 '18 at 16:22
  • $\begingroup$ ... by adding and removing Sex for example (full vs. null model) you can figure out whether one model accounts for more variation in the data than the other but not whether the effect Sex is significant in itself. For example you could add another covariate (or factor) to the model that increases the explanatory power of the model but it may not show up significant in itself when you look at it in a ANOVA-type table. $\endgroup$ – Stefan Feb 10 '18 at 16:28
  • $\begingroup$ @Stefan thanks for the comments. I am not quite sure about the term explanatory powe. As far as I realize the full mode is different from the null by only Sex. Then the only source of difference in models is sex. as a result anova reflects the effect of sex. On the other hand, as you truly pointed out, coefficient p-vale reflect the relative effect of Sex for Females compare to the baseline (male+mean response and mean age). Now the question is: how we should define the effect of sex (is it relative to other terms or only single term as in car:::Anova). I am still looking for good answer:) $\endgroup$ – TPArrow Feb 10 '18 at 16:54
  • $\begingroup$ By explanatory power I mean how well the fitted model can explain the underlying data, that is which predictor combination leaves the least amount of unexplained variation in the model. Figuring out this specific set of predictor variables is what I meant by model selection. anova() can be used to do that, i.e. compare competing models relative to each other with respect to variation explained. ... $\endgroup$ – Stefan Feb 10 '18 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.