12
$\begingroup$

Researchers often use two measures that have very similar items and argue that they measure different things (e.g., "I always worry when I am around cars"; "I am fearful of cars"). Lets call the hypothetical measures the Fear of Cars Measure and Anxiety from Automobiles Scale. I am interested in testing empirically if they indeed assess different latent constructs, or if they measure the same thing.

The two best ways I can think to do this would be through exploratory factory analyses (EFA) or confirmatory factor analysis (CFA). I think EFA would be good because it allows all of the items to load freely without constraints. If items from the two scales load on the same factors, then I can conclude that the measures likely don't assess different things very well. I can also see the benefits in CFA, however, since I will be testing pre-defined models. For example, I could compare the fit of a model in which all items load onto a single factor (i.e., they don't assess different constructs) or the items are separated into the expected measures. An issue with CFA, I suppose, is that it would not really consider alternative models (e.g., a three factor model).

For the purposes of discussion, lets also perhaps consider that there may be two other very similar measures out there (e.g., Car anxiety questionnaire and the Scales for the assessment of car fears) that I wish to throw into the mix!

How can I best statistically determine if two measures assess different constructs?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ An alternative to {E|C}FA is the multi-trait multi-method approach. This is basically founded on a correlational approach--with its pros and cons (wrt. latent trait)--and it has been discussed on the following threads, among others: stats.stackexchange.com/a/9944/930; stats.stackexchange.com/q/24418/930. $\endgroup$
    – chl
    Jul 22, 2012 at 21:02
  • $\begingroup$ Yes, that would be quite an interesting approach! Unfortunately, we only use one method in this area usually (e.g., individual self-report questionnaires). $\endgroup$
    – Behacad
    Jul 23, 2012 at 2:59
  • 2
    $\begingroup$ The MTMM technique can be used with self-reported measures collected on two different instruments assessing closely related or similar constructs. Alternative approaches include more elaborated factor-analytic methods and structural equation modeling. $\endgroup$
    – chl
    Jul 23, 2012 at 7:52
  • 1
    $\begingroup$ There are many papers available, including this review Structural equation modeling of multitrait-multimethod data: different models for different types of methods, or this paper Analysing multitrait–multimethod data with structural equation models for ordinal variables applying the WLSMV estimator which shows the general idea. I can try to find a better reference for the context of your study, though. Could you tell us: whether items are ordinal (e.g., Likert-type) or binary, sample size, and the number of facets you want to assess? $\endgroup$
    – chl
    Jul 23, 2012 at 16:01
  • 1
    $\begingroup$ Thanks for your reply! The questionnaires are Likert-type (5 options usually, but perhaps some have 4). There are probably 4 or 5 questionnaires that may or may not assess the same thing, and I am curious to test this empirically. I have a sample of maybe 300 now. As for number of facets, I am not sure what you mean exactly (factors?), but each measure could theoretically assess different things (so 4-5 different factors), or they assess the same thing (1 factor), or anything in between! Would MTMM be a good way to determine if they assess different latent constructs? $\endgroup$
    – Behacad
    Jul 23, 2012 at 17:23

2 Answers 2

Reset to default
3
$\begingroup$

These methods are examples of application of exploratory and confirmatory data analysis. Exploratory data analysis looks for patterns while confirmatory data analysis does statistical hypothesis testing on proposed models. It really should not be viewed in terms of which method to use it is more a matter of what stage in the data analysis you are at. If you are unsure of what factors to include in your model you apply EFA. Once you have eliminated some factors and settled on what to include in your model you do CFA to test the model formally to see if the chosen factors are significant.

Improve this answer
$\endgroup$
3
  • 5
    $\begingroup$ Thank you for the reply, although I feel you have not quite answered the question. I appreciate the differences in EFA and CFA and how they answer different questions, I am simply wondering which may be most appropriate in this context. Given you answer, I am inclined to think you are suggesting EFA. $\endgroup$
    – Behacad
    Jul 22, 2012 at 17:29
  • $\begingroup$ Do you have a scoring measure for each and do you give both surveys to the same individuals. I am thinking that you could pair the scores and look to see if there is high correlation. $\endgroup$
    – Michael R. Chernick
    Jul 22, 2012 at 18:14
  • $\begingroup$ All participants will complete all of the questionnaires. I am not sure what you mean by a "scoring measure". I will simply be summing the scores on all the questions within the questionnaire. $\endgroup$
    – Behacad
    Jul 22, 2012 at 19:09
1
$\begingroup$

If I understand your question correctly it is a question about testing. Then simply testing requires a kind of confirmatory factor-analysis, the same as the question: "do the means in the subgroups really differ?" requires a t-test.

Unfortunately(?) with the selection of the general approach of the appropriate method of factor analysis also different mathematical (and statistical) models are often implied, for instance, if you select "CFA" in SPSS then it is implied that you assume uncorrelated errors and that uncorrelated errors are estimated and the estimation is excluded from the model - so, in my opinion, because of the further implications the initial selection of the correct factor analytic approach is often compromised by this mathematical/statistical implications.

In short: your question is one of the sort "testing the null", thus you need CFA or better: the methods developed in the framework of SEM (structural equation modelling). Note, there is a friendly and helpful mailing list full of experts in SEM called "SEMNET" and since I'm not a real expert you might refine your feedback by asking there...

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your reply. I am familiar with CFA, EFA, and SEM, but I am not sure how to specifically explore if two questionnaires essentially measure the same thing. How would you propose I do this in SEM? $\endgroup$
    – Behacad
    Jul 24, 2012 at 17:37
  • $\begingroup$ @Behacad: I'd go and ask in SEMNET :-) Well, in fact I've no experience with the coefficients for testing of latent structures. Possibly a good introduction is given by a book of James Steiger, to whom SEMNETters often refer. (Sorry I cannot be of more help here) $\endgroup$
    – Gottfried Helms
    Jul 25, 2012 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.