A well established questionnaire measuring interpersonal relations is modified to measure interorganizational relations. 11 parameters of the relation are determined by 4 items each. Items are removed until the best possible $\alpha$ per scale is achieved (.58 - .81). This results in some scales with 3 items, some scale with 4 items.
A confirmatory factor analysis is applied to calculate the estimates of the items and verify the internal structure of the questionnaire.
As expected with some $\alpha$ as low as .58 and .61 the CFA does not fit very well. Since it is "kind of" a new scale, is it ok to use the predictions for further modelling and report them?
The fit indices are (bad):
Number of observations 251 Estimator ML Robust Minimum Function Test Statistic 2475.330 2029.585 Degrees of freedom 854 854 P-value (Chi-square) 0.000 0.000 Scaling correction factor 1.220 for the Satorra-Bentler correction Model test baseline model: Minimum Function Test Statistic 5718.372 4583.881 Degrees of freedom 946 946 P-value 0.000 0.000 Full model versus baseline model: Comparative Fit Index (CFI) 0.660 0.677 Tucker-Lewis Index (TLI) 0.624 0.642 Loglikelihood and Information Criteria: Loglikelihood user model (H0) -18173.075 -18173.075 Loglikelihood unrestricted model (H1) -16935.410 -16935.410 Number of free parameters 180 180 Akaike (AIC) 36706.150 36706.150 Bayesian (BIC) 37340.731 37340.731 Sample-size adjusted Bayesian (BIC) 36770.110 36770.110 Root Mean Square Error of Approximation: RMSEA 0.087 0.074 90 Percent Confidence Interval 0.083 0.091 0.070 0.078 P-value RMSEA <= 0.05 0.000 0.000 Standardized Root Mean Square Residual: SRMR 0.146 0.146
The real question is probably not only whether to continue with estimates based of a bad fit, but rather whether the scale is new or not.
Barrett (2007) recommends to not rely on "approximate fit indices" at all.
The criterion used for "fit" is actually an abstract concept in the majority of SEM models. It is clearly not predictive accuracy. In fact whether models "approximately fit" with an RMSEA of 0.05 or 0.07 is a literally meaningless scientific statement.
Barrett, P. Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 2007, 42, 815-824 [PDF]