I have translated and adapted a questionnaire (based on a 5 factor model; 23 items) into my native language by following ITC and MAPI guidelines. Forward / backward translation, expert panel review, cognitive debriefing followed review again and pilot testing with bilingual sample.
Initially I collected data from a sample 150 students (9-13 years old) and ran a PCA with varimax rotation. It yielded an unclear picture. Later I found a significant Shapiro-Wilk value indicating distribution was non normal. Based on these findings and considering the correlation among latent variables, I computed an EFA on same data using principal axis factoring with promax rotation. It provided a clearer picture than other methods. The factor matrix provided a 2 factor solution and the pattern matrix yielded 5 factors; two of which had 2 items each, and the other two had 8 and 5 items respectively. All items had reliabilities of more than .60. Four items were not loaded on a distinct factor. Thus, I retained 19 items.
I collected another sample for main study afterwards (N = 672), and conducted CFA. I tried three models:
Original 5 factor model: CMIN=414.77, DF= 199 (p< .001); CMIN/DF 2.08; RMR = ,o6, GFI= .94, IFI= .92; TLI = .908; CFI= .92; RMSEA.04
It was apparently a good model fit yet two of the factors showed poor reliabilities (less than .55).
A two factor model (which is mostly retained in other translations of this scale (i.e., Italian); and is supported by the factor matrix in our previous study). Values were: CMIN=467.28, DF= 208 (p< .001); CMIN/DF 2.24; IFI= .90; TLI = .89; CFI= .90; RMSEA.04 (reliabilities were .84 for F1 and .57 for F2). Yet two items in F2 had to be deleted due to poor item-total correlation (less than .07). But if I delete them, F2 will contain 2 items only .55 reliability and F1 has 17 items.
Based on the pattern matrix of principal axis factoring, rotated solution, we ran another model (4 factor) resulting in good fit indices similar to model 1 values crossed .90 with RMSEA .04. Yet again, only 2 factors had better reliabilities. However, factor 2 and 4 covariance was .94 so I later merged these factors into one and eliminated items on factor 3 (that emerged as F2 in the previous model) because it always yielded low reliability (.55-.57, etc.) and was based on reverse scored items. The analysis was rerun. This time it provided an even better fit with values up to .93 and .94 for NFI, and CFI; RMSEA= .05. However covariance between these two factors was still high (.84).
Did I follow the right track or I should have tested the model obtained from EFA only? Should I converge the two factors and try a unidimensional model or should I compute EFA once again?