2
$\begingroup$

I have a question regarding fit indices for a cfa based on 2 groups/samples. Specifically, I have fitted a cfa model with one factor based on 7 indicators (questionnaire items), and two samples (clinical vs. non-clinical), with the restriction of equal factor loadings in both samples. I am using the lavaan package in R, using the following code to fit the model:

fit <- cfa (Model, data = data, group = sample, group.equal = c("loadings"))

The summary of fit measures provides me with two different chi-square values for the two samples, but only one value for supplemental fit measures including

  • the RMSEA
  • SRMR
  • CFI
  • TLI.

If there are separate chi-square values for both samples, should I not also get supplemental fit measures for both samples? And which model/sample do supplemental fit measures in the output refer to?

My understanding was that supplemental fit indices (RMSEA, CFI....) are ultimately based on the chi-square and I am struggling to understand why there are separate chi-square values by (sub-)sample and just one value for the entire sample for the supplemental fit indices. Is there a way to get either an individual chi-square value for the whole sample, or supplemental fit indices by group?

$\endgroup$

1 Answer 1

1
$\begingroup$

When you are obtaining a summary of your model using the summary(fit) function try adding summary(fit, fit.measures=T). You should obtain additional information about common fit indices for each group in your output.

UPDATE:

I was flat out incorrect about what the fit.measures=T argument does. I should have checked before posting to confirm. It provides overall model fit indices in the summary() output, and does not split said indices by group when performing a multigroup analysis.

And perhaps more importantly, I should have thought about the answer at a more fundamental level as the other fit indices you mention are not appropriate to calculate per group solely within a multi-group model. Take for instance CFI:

$$ CFI = \frac{[\chi_{null}^2-df_{null}]-[\chi_{model}^2-df_{model}]}{[\chi_{null}^2-df_{null}]}$$

where $\chi_{null}^2$ is the $\chi^2$ statistic from the baseline model and $\chi_{model}^2$ is the $\chi^2$ for your obtained model.

Even if you were to take the $\chi^2$ statistic for a particular group (which we know we can obtain from the output), how, in the context of a multi-group model would you be able to obtain the correct $\chi_{null}^2$, for that group? After all the baseline $\chi^2$ is based on all your observations in a multigroup model.

For multi-group SEM models in lavaan and LISREL (and I imagine other SEM programs) it would appear that the optimal strategy is to run your analysis separately for each group, which to be fair is a good practice to determine, initially, the degree to which your proposed model actually fits your obtained data in each group. You can obviously obtain model fit indices separately for your groups in this manner.

And, provided that you do not add any constraints to your subgroup analyses, you should obtain model fit $\chi^2$ statistics for your models that correspond to a completely unconstrained multigroup model (i.e., the loadings, intercepts, residuals, etc. are all free to vary).

Now I do wonder whether you could calculate something like a modified CFI per group by hand using the baseline $\chi^2$ from your subgroup analyses. Using this approach you might try the following:

$$CFI = \frac{[\chi_{null|g1}^2-df_{null|g1}]-[\chi_{model|g1}^2-df_{model|g1}]}{[\chi_{null|g1}^2-df_{null|g1}]}$$

where $\chi_{null|g1}^2$ is the baseline $\chi^2$ from the separate subgroup analysis for group 1 and $\chi_{model|g1}^2$ is the $\chi^2$ statistic for group 1 derived from your multigroup model. Finally, $df_{model|g1}$ is based on the number of free parameters that can be estimated in the multigroup model for group 1 alone.

I imagine it might be possible to extend this approach to other fit indices. For instance you could adjust RMSEA as follows:

$$RMSEA = \sqrt{\frac{\chi_{model|g1}^2-df_{model|g1}}{df_{model|g1}(n_{g1}-1)}}$$

At first blush something along these lines seems defensible, though I am not sure how often these kinds of supplementary analyses are performed or reported. Also I imagine specifying the correct degrees of freedom can be tricky as you are not really fixing estimates to a particular value. Instead, you are fixing them to be equal between two or more groups, and the target group contributes to the model's final estimate. Perhaps, such "side" analyses have diagnostic value in the model building phase, but I would not confidently report them alongside traditionally computed fit indices calculated in more standard ways.

Also, having spent a lot more time thinking about this problem now in general, I am curious as to the underlying motivation for wanting separate fit indices by group within the context of your multigroup model. It would seem to me that what matters most is the degree to which such a model, including your constraints of interest, can effectively reproduce the observed data in your multiple groups.

$\endgroup$
2
  • $\begingroup$ Thanks for the suggestion. I had actually done exactly what you suggested - added the "fit.measures = TRUE" (i had originally added that code to my question, but it seems to have been edited out), but it just gave me one value for RMSEA, CFI and TLI - I am still unsure what it refers to - whereas it gave me separate values by sample for the chi-square and factor loadings. Thank you though $\endgroup$
    – Birke
    Jan 17, 2018 at 17:50
  • $\begingroup$ I have updated my response after some additional thought. If figuring out a way to calculate fit indices by group is particularly relevant for your research within the context of your multigroup models you may want to spend some time digging through the Mplus message boards. Many of those threads get very technical and can offer insights into these sorts of non-obvious problems. $\endgroup$ Jan 17, 2018 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.