EFA clearly supports one-factor, measure is internally consistent, but CFA has poor fit?

Question

I am exploring the psychometric properties of a 10-item self-report measure. I have about 400 cases in two independent samples. The items are completed on 4-point Likert scales. An EFA clearly supports a one-factor solution (e.g., first eigenvalue over 6, all others under 1) and Cronbach's alpha is good (e.g., .90). No item has low item-total correlation.

I originally wanted to do a CFA (EFA was just a follow-up after I saw CFA wasn't good) testing a one-factor model. To my surprise, fit for the model was relatively poor:

CFI=.91
TLI=.88
RMSEA=.13

Moreover, loadings for each of the items is quite good (.65+).

Oddly, the SRMR=.05, which is acceptable / good.

Modification indices suggest I correlate errors all over the place. If there was a clear rational to do so (e.g., some of the items have very similar wording) I would do this; however, all the measures are worded similarly, and correlating all the error terms would be odd and painful.

I have never seen a case like this. The measure is internally consistent and is clearly comprised of one factor in EFA, but it exhibits poor fit in the CFA. The results are congruent in both independent samples (from different continents). I tried a two-factor CFA (grouped 5 random items) and fit was the same, or even marginally better.

Here are my questions:

1. Why is the fit according to CFI/TLI/RMSEA so poor given the EFA/Cronbach alpha/factor loadings?
2. Why is the SRMR good while the other indices are not? I know they measure different things, but in my experience, they almost always converge.
3. Should I correlate some of the errors?

Example items:

• You have thoughts about your shortcomings
• You have thoughts that are difficult to forget
• You think about the situation all the time

Answer 1

That's pretty normal.

CFA is a much more stringent criterion than EFA. EFA attempts to describe your data, but CFA tests if the model is correct.

One reason for non-convergence is low average correlations (but then I'd expect RMSEA to be better). The chi-square test is essentially a test that your residuals are equal to zero, and RMSEA, TLI and CFI are transformations of the test.

Fit is always going to be better in a two factor solution than a one factor solution (they're nested).

Some more questions: What was your sample size? What's the average correlation? What's chi-square and df, what's the chi-square of the null model?

Should you add correlated errors? Perhaps, but when you do that you are introducing additional factors. With a fit like this you might need to add a lot, and then you end up with a mess - it's best if they are justified in some way. For example, your second and third items are about intrusive thoughts - that could be a justification.