1
$\begingroup$

The analysis consists in comparing the means for 6 phonological constraints to the level of indifference, defined as 3.5. The same participants provided judgments for all the 6 constraints, which makes it a repeated measures design. I basically need to establish if subjects are sensitive to any of these constraints. If they are, then the means should be significantly different from 3.5. The code that was advised to me is:

summary( lme(offsetVal ~ 0 + constraint, random=~1|subject, data=FRdata) )

Where the offsetVal is a number that you get when subtracting the 'indifference level' (3.5) from the means for each constraint from each participant.

OUTPUT:

Linear mixed-effects model fit by REML
 Data: FRdata 
       AIC      BIC    logLik
  226.4168 246.4153 -105.2084

Random effects:
 Formula: ~1 | subject
        (Intercept)  Residual
StdDev:   0.1704487 0.6919447

Fixed effects: offsetVal ~ 0 + constraint 
               Value Std.Error DF    t-value p-value
constraint1 -0.35000 0.1781573 75 -1.9645562  0.0532
constraint2 -0.20000 0.1781573 75 -1.1226036  0.2652
constraint3  0.29375 0.1781573 75  1.6488240  0.1034
constraint4  0.08125 0.1781573 75  0.4560577  0.6497
constraint5  0.36875 0.1781573 75  2.0698003  0.0419
constraint6 -0.03750 0.1781573 75 -0.2104882  0.8339
 Correlation: 
            cnstr1 cnstr2 cnstr3 cnstr4 cnstr5
constraint2 0.057                             
constraint3 0.057  0.057                      
constraint4 0.057  0.057  0.057               
constraint5 0.057  0.057  0.057  0.057        
constraint6 0.057  0.057  0.057  0.057  0.057 

Standardized Within-Group Residuals:
        Min          Q1         Med          Q3         Max 
-3.30462921 -0.42615935  0.04149831  0.53233941  2.33326486 

My editor is saying that p-values that seem to be significant (for constraint 1 and constraint 5) will not be significant if a correction for multiple t-tests is applied. My question is: Do I need this correction and how to apply it?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

Strictly speaking, yes, because you test 6 times in the same model (family of tests). However, convention often is that in a given regression model, multiple testing correction is not applied (maybe it should be).

If you apply a correction you adjust the significance level $\alpha$. This is done differently depending on the correction procedure you choose. A common but conservative procedure is called Bonferroni adjustment. This adjustment asks for $\alpha/m$ where $m=6$ tests in your case. Since in your case you apparently use $\alpha=5 \%$ the adjusted level would become $0.833 \%$ and hence none of the tests would be significant.

There is one straight forward way how you can avoid the multiple testing issue. Instead of testing 6 times $H_{0i}: \beta_i=0$ where $i$ denotes hypotheses $i=1,...,6$, you can test one time simultanously $H_{0}: \beta_1=\beta_2=...=\beta_6=0$. For this you need to fit an empty mixed effects model with $lme$ and evaluate the difference in $AIC$ (or BIC, or twice the difference in log-likelihoods) of the models. The difference is Chi-quare distributed with six degrees of freedom. This is called a log-likelihood ratio test (LRT).

The obvious advantage of this LRT procedure is that you now only have one test and may use your desired significance level without 'adjustment'. The disadvantage is that this hypothesis test does not tell you anymore which of the contrasts is different from zero, but only that one or more of them are.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.