I am exploring the psychometric properties of a 10-item self-report measure. I have about 400 cases in two independent samples. The items are completed on 4-point Likert scales. An EFA clearly supports a one-factor solution (e.g., first eigenvalue over 6, all others under 1) and Cronbach's alpha is good (e.g., .90). No item has low item-total correlation.
I originally wanted to do a CFA (EFA was just a follow-up after I saw CFA wasn't good) testing a one-factor model. To my surprise, fit for the model was relatively poor:
CFI=.91
TLI=.88
RMSEA=.13
Moreover, loadings for each of the items is quite good (.65+).
Oddly, the SRMR=.05
, which is acceptable / good.
Modification indices suggest I correlate errors all over the place. If there was a clear rational to do so (e.g., some of the items have very similar wording) I would do this; however, all the measures are worded similarly, and correlating all the error terms would be odd and painful.
I have never seen a case like this. The measure is internally consistent and is clearly comprised of one factor in EFA, but it exhibits poor fit in the CFA. The results are congruent in both independent samples (from different continents). I tried a two-factor CFA (grouped 5 random items) and fit was the same, or even marginally better.
Here are my questions:
- Why is the fit according to CFI/TLI/RMSEA so poor given the EFA/Cronbach alpha/factor loadings?
- Why is the SRMR good while the other indices are not? I know they measure different things, but in my experience, they almost always converge.
- Should I correlate some of the errors?
Example items:
- You have thoughts about your shortcomings
- You have thoughts that are difficult to forget
- You think about the situation all the time