Can you compare a two-factor solution from CFA to a three-factor solution via Chi²-tests? I got a questionnaire with 30 items and our theory proposes two different factor solutions for this questionnaire. I want to run CFA based on this theory. Solution 1 would include 2 factors on which 15 items load, respectively. Solution 2 would include 3 factors on which different subsets of items would load.
My question is now whether these models would qualify as nested and I can thus compare their model-fit using Chi²-test or a likelihood ratio test.
Thanks!
 A: No.
For model B to be nested within model A, it has to be the case that model A can be made to be the same as model B by fixing some of A's parameters to a set value, typically 0 (removing a loading from CFA is equivalent to fixing it to 0).
What's more, you're not even using the same data for both models in this case, so they're definitely not nested!
A: Unfortunately these models are not nested. It is possible to test different factor structures using chi-square tests, but only if you are comparing models where one of the models splits a factor into more than one factor. For example, you can use a chi-square test to compare a 2-factor solution with Factor A comprising items 1-10 and Factor B comprising items 11-30, and a 3-factor solution with Factor A comprising items 1-10 Factor B* comprising items 11-20, and Factor C* comprising items 21-30. In this case, you are testing whether Factor B can be split into Factor B* and Factor C*. The reason these two models are nested is that the model with Factor B is equivalent to the model with Factors B* and C* when the correlation between Factors B* and C* is fixed a 1. Therefore, the test comparing the two models tests whether that restriction is valid.
Comparing models with completely different factor structures cannot be done using a chi-square test. You can use other methods of comparing models, like BIC, instead, though these do not provide formal tests.
