How should I adjust for the number of tests I do when using the chi-square difference test, where the null hypothesis of the test is that two models do not differ? I am using this test in the context of confirmatory factor analysis, when conducting measurement invariance evaluation. Here, the 'desired' outcome is that the p-value is insignificant - because I then have evidence to use a more constrained model.

I am a bit confused, because usually you intend to provide evidence for the alternative hypothesis. Here it is the other way around. Normally, if I would use the Bonferroni correction I could roughly divide the significance level by the number of tests.

If I do it this way, then the number of tests isn't really 'punishing' me, but rather making it easier to fail to reject the null hypothesis, which I intend to do.

EDIT: I am doing multiple tests to find out what level of measurement invariance holds, in the context of confirmatory factor analysis. Here, I start with a general model without restrictions and progress to a more restricted model in two or three steps, each time using a chi-square difference test.

Another context I plan to be using it in, is in longitudinal analysis where I model the covariance pattern (i.e. covariance pattern modeling), where I start with the most general structure (all covariances freely estimated). Then I would progress to more restricted / simpler model solutions, e.g.: autoregressive patterns, equal variances, ... etc.

  • $\begingroup$ Why are you doing multiple tests, I'm not sure I understand. $\endgroup$ – Jeremy Miles May 29 '18 at 16:56
  • $\begingroup$ I added two examples to the question. $\endgroup$ – Amonet May 29 '18 at 19:28
  • $\begingroup$ I wouldn't do any correction here. And I've never heard of anyone doing it. $\endgroup$ – Jeremy Miles May 30 '18 at 0:49
  • $\begingroup$ Thanks, although I find it somewhat counterintuitive. When I start from general model 1 and compare / test to submodel 1, then compare submodel 1 to submodel 2, ...., submodel 5 to submodel 6. That's multiple testing, isn't it? Or am I mistaken here, I'm asking since I am a bit confused :) $\endgroup$ – Amonet May 30 '18 at 8:44
  • $\begingroup$ It's a specific case of a more general case of testing a nested sequence of models. If you asked this question as a more general case (not about SEM), you might get good answers. $\endgroup$ – Jeremy Miles May 31 '18 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.