1
$\begingroup$

The last question I have posted here: Whether there is significant difference between two gender groups

There are 16 people in the dataset(using subjectID to identify them). Each year, when these people were ages 11, 12, 13, 14, and 15, they provided us a score y. The overview of this dataset is like this:

> df
     id Sex Age    y
1     9   F  11 2.23
2    45   M  11 1.12
3   268   M  11 1.45
4   314   F  11 1.22
5   442   F  11 1.45
6   514   M  11 1.34
7   569   F  11 1.79
8   624   M  11 1.12
9   723   F  11 1.22
10  918   F  11 1.00
11  949   M  11 1.99
12  978   M  11 1.22
13 1105   M  11 1.34
14 1542   F  11 1.22
15 1552   F  11 1.00

We now want to know whether there is a significant difference between male and female people in the change in their y through time. I tried to use Mixed Effect Models to do it:

> fit_full <- lmer(y~Age:Sex+(Age|id), data = df,
               control = lmerControl(optimizer ="Nelder_Mead"),REML = F)

> fit_null <- lmer(y~Age+Sex+(Age|id), data = df,
               control = lmerControl(optimizer ="Nelder_Mead"),REML = F)

> anova(fit_null,fit_full)

I got an output:

> anova(fit_null,fit_full)

Data: df
Models:
fit_null: y ~ Age + Sex + (Age | id)
fit_full: y ~ Age:Sex + (Age | id)
         Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)    
fit_null  7 77.840 94.515 -31.920   63.840                             
fit_full  7 77.745 94.419 -31.872   63.745 0.0956      0  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Could I say we have enough evidence to say there is a significant difference between male and female people in the change in their y through time?

$\endgroup$
  • 1
    $\begingroup$ I think that probably you full model should be fit_full <- lmer(y ~ Age * Sex + (Age | id), data = df, control = lmerControl(optimizer = "Nelder_Mead"), REML = FALSE). $\endgroup$ – Dimitris Rizopoulos Mar 28 at 8:13
  • $\begingroup$ I am confused about how to write the null hypothesis of such nested model test:y ~ Age*Sex + (Age|id) and y ~ Age+Sex + (Age|id)? $\endgroup$ – tieguanyin Mar 28 at 23:41
1
$\begingroup$

Short answer: No, you cannot say that there is "evidence of a significant difference". Your test shows that there is statistically significant evidence of a non-zero difference in the slopes for men and women, which is a different thing.


Longer answer: Firstly, it is important to get the terminology of hypothesis testing correct. Classical hypothesis tests do not measure any such thing as a "significant difference" between groups. Statistical "significance" in this context is a measure of the magnitude of evidence in favour of the alternative hypothesis of a non-zero difference between the unknown slopes for these two groups. The "significance" here refers to the magnitude of evidence of a difference, not the magnitude of the difference itself. Hence, you should refer to there being "statistically significant evidence of a difference" --- not a "significant difference" in the slopes.

In this particular case, you have conducted an ANOVA test of the nested models, where the null hypothesis is that there is no interaction term (i.e., zero difference in the slopes for males and females) and the alternative hypothesis is that there is a non-zero interaction term (i.e., a non-zero difference in the slopes for males and females). This test gives a p-value of $2.2 \times 10^{-16}$, which is extremely low, and well below any significance level that would be likely to be chosen for testing. Assuming that this nested model form is reasonable (e.g., by checking diagnostic plots, etc.), you have indeed identified statistically significant evidence of a difference in the slopes between the male and female groups.

In regard to this issue, it is again important to stress that there is a huge difference between "significant evidence of a difference" versus "evidence of a significant difference". These are two very different things, and conflating these two things is an example of the quantifier-shift fallacy. It is the same error as confusing "significant evidence of a crime" (e.g., the police collect strong evidence that a motorist ran a stop-sign) and "evidence of a significant crime" (e.g., evidence of arson, rape, or murder).

$\endgroup$
  • $\begingroup$ I think the models are not nested, the degrees of freedom are the same in the two models. $\endgroup$ – Dimitris Rizopoulos Mar 28 at 5:31
  • $\begingroup$ Although I wrote the R command, but I don't know how to write the null hypothesis of the test? $\endgroup$ – tieguanyin Mar 28 at 22:38
  • $\begingroup$ Having looked at your R commands, I think Dimitris is correct, that you have not used nested models. Full model should be y ~ Age*Sex + (Age|id), not y ~ Age:Sex + (Age|id). $\endgroup$ – Ben Mar 28 at 23:24
  • $\begingroup$ Thanks Ben. And now I am confused about how to write the null hypothesis of such nested model test:y ~ Age*Sex + (Age|id) and y ~ Age+Sex + (Age|id)? $\endgroup$ – tieguanyin Mar 28 at 23:40
  • $\begingroup$ Once you fix up the full model, your null hypothesis is $H_0: \beta_\text{Age:Sex} = 0$ and your alternative hypothesis is $H_0: \beta_\text{Age:Sex} \neq 0$. $\endgroup$ – Ben Mar 29 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.