2
$\begingroup$

A well established questionnaire measuring interpersonal relations is modified to measure interorganizational relations. 11 parameters of the relation are determined by 4 items each. Items are removed until the best possible $\alpha$ per scale is achieved (.58 - .81). This results in some scales with 3 items, some scale with 4 items.

A confirmatory factor analysis is applied to calculate the estimates of the items and verify the internal structure of the questionnaire.

As expected with some $\alpha$ as low as .58 and .61 the CFA does not fit very well. Since it is "kind of" a new scale, is it ok to use the predictions for further modelling and report them?


The fit indices are (bad):

  Number of observations                           251

  Estimator                                         ML      Robust
  Minimum Function Test Statistic             2475.330    2029.585
  Degrees of freedom                               854         854
  P-value (Chi-square)                           0.000       0.000
  Scaling correction factor                                  1.220
    for the Satorra-Bentler correction

Model test baseline model:

  Minimum Function Test Statistic             5718.372    4583.881
  Degrees of freedom                               946         946
  P-value                                        0.000       0.000

Full model versus baseline model:

  Comparative Fit Index (CFI)                    0.660       0.677
  Tucker-Lewis Index (TLI)                       0.624       0.642

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)             -18173.075  -18173.075
  Loglikelihood unrestricted model (H1)     -16935.410  -16935.410

  Number of free parameters                        180         180
  Akaike (AIC)                               36706.150   36706.150
  Bayesian (BIC)                             37340.731   37340.731
  Sample-size adjusted Bayesian (BIC)        36770.110   36770.110

Root Mean Square Error of Approximation:

  RMSEA                                          0.087       0.074
  90 Percent Confidence Interval          0.083  0.091       0.070  0.078
  P-value RMSEA <= 0.05                          0.000       0.000

Standardized Root Mean Square Residual:

  SRMR                                           0.146       0.146

Edit:

The real question is probably not only whether to continue with estimates based of a bad fit, but rather whether the scale is new or not.


Edit2:

Barrett (2007) recommends to not rely on "approximate fit indices" at all.

The criterion used for "fit" is actually an abstract concept in the majority of SEM models. It is clearly not predictive accuracy. In fact whether models "approximately fit" with an RMSEA of 0.05 or 0.07 is a literally meaningless scientific statement.


Bibliography

Barrett, P. Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 2007, 42, 815-824 [PDF]

$\endgroup$
3
$\begingroup$

Some immediate comments:

1) I would not predict from this model, since it has bad fit (CFI should be >.95 and RMSEA <.05).

2) It appears you are exploring and confirming on the same data set. This is capitalizing on chance and generally not recommended. It is surprising that your CFA still has bad fit. If you explore on the same data, CFA fit often is much better.

3) You could try exploratory FA or stepwise CFA with modification indices (Lagrange multipliers) to find a better fitting model. I would do this instead of relying on a bad index like $\alpha$ (it is not a good estimator of scale reliability, in fact it is just one estimator, which may be too optimistic about reliability).

4) In any way, I would recommend cross-validating your model. This means splitting your data set in two, explore on one (EFA, modification idices), and confirm (CFA) on the other. This way you avoid capitalizing on chance in scale exploration.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer. The reason I'm conducting a CFA and not start with an EFA as one would with a completely new scale, is because a very similar instrument is extremely well established in the literature - in a slightly different context. On the other hand the data clearly show that the new application must differ more than expected. I'm just not sure how to proceed. Ideas are welcome. $\endgroup$ – Roman Jan 8 '14 at 17:16
  • $\begingroup$ I would try exploratory analyses followed by CFA to see if you find a scale that fits your data and then judge the difference to the established scale. If you do not have an exact reproduction of an established scale, it is hard to say exactly why it does not fit your data. $\endgroup$ – tomka Jan 8 '14 at 17:20
  • $\begingroup$ The parallel FA suggests only eight factors instead of eleven. Eleven factors however explains still 50-55 percent of the variation. However due to the nature of the method some of the item don't load on the factor they should. Hence the fit gets worse with a CFA. I'm struggling to define new scales due to the similarity of the scales to a existing and well established instrument. What a dilemma. :-/ $\endgroup$ – Roman Jan 9 '14 at 10:53
  • 2
    $\begingroup$ @tomka makes some good points, but I have some additional comments. Look at the residual correlation matrix. Fit statistics in SEM can be misleading, because they only quantify model mis-specification in one value but do not tell you where it is. Both this, and looking at which items have large error variances, is a great way to detect problematic items. And to add another question, what is your sample size? $\endgroup$ – dmartin Jan 9 '14 at 14:53
  • 1
    $\begingroup$ @dmartin is right. In your particular data there may be a somewhat different error structure, e.g. correlated errors for some of the questions. This happens for example due to order effects in non-randomized item batteries. By adding some correlated errors to the model, after inspecting the residual cov. matrix, you may achieve much better fit of the established scale model. $\endgroup$ – tomka Jan 10 '14 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.