1
$\begingroup$

For my thesis I am conducting a factor analysis of a Belgian personality questionnaire, using the lavaan package for R. I have applied a split-sample procedure, and use sample 1 for exploratory factor analysis (EFA), and sample 2 for confirmatory factor analysis (CFA). Both samples have N > 300. All models below have a two-factor solution.

However, when I test the model that I identified in EFA (sample 1) using CFA (sample 2), I get a poor fitting model (low CFI & TLI, high RMSEA). On the other hand, several modification indices are suggested that seem to make sense (i.e., let several error terms of similar items correlate), after which I do get a good fitting adjusted model. This is indicated by a CFI and TLI of .95, and an RMSEA of .05.

Now, my main problem is that when I compare the fit of my Belgian model with the fit of a previously identified factor structure of the English version of the questionnaire (tested on the same data), that the AIC of this English model is lower (7500ish vs 8500ish) than my adjusted model. This makes sense as my new model is more complex and AIC controls for this. However, the overall fit of the English model is not very high (CFI/TLI .89ish, RMSEA .07ish). Thus, although the English model has a lower AIC and thus is the preferred model, it fits rather poorly on the current data.

I want to use the factors (subscales) I identified in a future study in which I relate these to behavioral measures, but I don't have time for another factor analysis to test any model of the questionnaire again. Should I continue using my own Belgian model/subscales, or is it better to use the English model/subscales instead? Also, is it commonly accepted to use subscales from a questionnaire from a different language if this better fits the data than native-language questionnaires?

I hope this somewhat makes sense - thanks for any replies!

Sjon

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

AIC can't be used to to compare models fitted on different datasets, so comparing your model to that of the English model makes no sense whatsoever. So, there is no justification at all to abandon your model and favour the English one on this basis.

I would also advise a little caution in the use of modification indices, which allow the analyst to obtain a better fitting model without thinking. However, it does appear from what you said that these residual error covariances are justified in your case, since you say that they are similar items.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you for your reply! I'm afraid I did not explain this very well. I used my Sample 2 data for CFA. On this, I plotted my Belgian factor structure AND the English factor structure. So I compared the fit indices of these two models on the SAME data. In this case, the English solution has a lower AIC, but the general fit of this English solution on my dataset is moderate at best. $\endgroup$
    – Sjon
    Commented Aug 3, 2016 at 10:43
  • $\begingroup$ Oh I see. Well, if CFI, TLI and RMSEA all favour one model, I would prefer that. Also, BIC is preferable to AIC in this situation but I doubt it will make much difference. What do the model chi-square tests say ? $\endgroup$
    – Robert Long
    Commented Aug 3, 2016 at 10:51
  • $\begingroup$ BIC indeed goes the same direction. Regarding Chi squares: Belgian: Chi² (61) = 108.43, p < .001 English: Chi² (53) = 143.17, p < .001 $\endgroup$
    – Sjon
    Commented Aug 3, 2016 at 10:58
  • $\begingroup$ Neither model fits very well, but if these are your only 2 models I would choose the Belgian one. But if you have any other options to improve model fit I would explore them first. Was the factor structure you identified in EFA clearly the best one ? $\endgroup$
    – Robert Long
    Commented Aug 3, 2016 at 11:09
  • $\begingroup$ Yeah pretty much. There was a third factor with a higher eigenvalue than produced in parallel analysis, but it consisted of very few items and the scree plot showed best fit for a two-factor solution. I have examined that model on my CFA dataset but it fitted far worse than other models. I did notice that if I conduct EFA on my CFA dataset (sample 2), I get a somewhat different component solution than produced by my original EFA dataset (sample 1). Is it possible that due to random splitting of my sample I just ended up with two halves of the data that don't compare very well? $\endgroup$
    – Sjon
    Commented Aug 3, 2016 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.