8
$\begingroup$

I've been given a set of 20 Likert-items (ranging from 1-5, sample size n = 299) within the field of organizational research. The items are intended to measure a latent concept which is multidimensional, multifaceted and heterogenous in it's very nature. The goal is to create a scale(s) which can be nicely used to analyse different organizations and be used in logistic regression. Following the american association of psychology, a scale should be (1) unidimensional, (2) reliable and (3) valid.

Therefore we decided to select four dimensions or subscales with 4/6/6/4 items each; which are hypothesized to represent the concept.

The items werde constructed using the reflective approach (generating lots of possible items, and iteratively deleting items using cronbach's alpha and concept representation (validity) in three subsequent groups).

Using the available data, a preliminary parallel explanatory factor analysis based on polychoric correlations and using varimax rotation revealed that items load on other factors than expected . There are at least 7 latent factors as opposed to four hypothesized ones. The average inter-item correlation is quite low (r=0.15) albeit positive. The cronbach-alpha coefficient is also very low (0.4-0.5) for each scale. I doubt that an Confirmatory factor analysis would yield a good model fit.

If two dimensions were dropped, the cronbachs alpha would be acceptable (0.76,0.7 with 10 items per scale, which still could be made bigger by using the ordinal version of cronbachs alpha) but the scales themselves would still be multidimensional!

As I'm new to statistics and lack the appropriate knowledge, I am at a loss on how to proceed further. As I am reluctant to discard the scale(s) completely and resign to a descriptive-only approach, I've got different questions:

I) Is it wrong to use scales which are reliable, valid but not unidimensional?

II) Would it be appropriate to interpret the concept afterwards as formative and use the vanishing tetrad test to assess model-specification and use partial least squares (PLS) to arrive at a possible solution? After all, the concept seems to be more a formative than a reflective one.

III) Would using the item response models (Rasch, GRM etc) be of any use? As I've read, the rasch-models etc. also need the assumption unidimensionality

IV) Would it be appropriate to use the 7 factors as new "subscales"? Just discard the old definition and use a new one based on factor loadings?

I'd appreciate any thoughts on this one :)

EDIT: Added factor loadings & correlations

> fa.res$fa
Factor Analysis using method =  ml
Call: fa.poly(x = fl.omit, nfactors = 7, rotate = "oblimin", fm = "ml")

Factor loadings calculated from factor pattern matrix and factor intercorrelation matrix, only values above 0.2 are displayed

Factor Loadings

Improve this question
$\endgroup$
7
  • 3
    $\begingroup$ One suggestion I can make is that you should not use varimax rotation (or any other rotation that assumes orthogonal factors), it is much better to use an oblique rotation, such as direct quartimin. First, it is not clear why the underlying factors must be orthogonal; second, oblique methods allow you to estimate the factor correlations; and third, if the factors are orthogonal, an oblique rotation will return fundamentally the same results as varimax. On the other hand, if the underlying factors are not orthogonal, then varimax can yield dangerously flawed results. $\endgroup$
    – gung - Reinstate Monica
    Jan 15, 2012 at 17:52
  • $\begingroup$ Good question and you seem to be on a "good track." Why don't you post your factor loadings and inter-factor correlations after following the smart advice @gung has given. That should help people size up the dimensionality of your set of items. (But what do you mean by "parallel explanatory factor analysis"? Maybe exploratory factor analysis using parallel analysis to choose the number of factors?) By the way, it is the scales you will create that are "Likert scales"; it's not correct to call items "Likert items" unless they have been put through something like the process you've described. $\endgroup$
    – rolando2
    Jan 15, 2012 at 18:14
  • 3
    $\begingroup$ Another consideration is that there is typically a tradeoff between a scale's internal consistency (as measured by Cronbach's alpha) and its validity (as indicated by the breadth of the topics it covers). Perfect internal consistency is of course not desirable, as it means we have wasted our time dealing with alot of redundant questions. And overly broad coverage means the scale measures too many things, and none of them thoroughly. APA guidelines probably lean too far toward favoring internal consistency, but you have to make your own case for your own preferred tradeoff. $\endgroup$
    – rolando2
    Jan 15, 2012 at 18:30
  • $\begingroup$ @rolando2 Yes I meant exploratory factor analysis using parallel analysis to choose the number of factors! $\endgroup$
    – Jack Shade
    Jan 15, 2012 at 20:40
  • 2
    $\begingroup$ No problem. It sounds like @rolando2 has more experience with these issues than I do, and it looks like you're on your way. I should let him guide you further, but I will mention one more thing prompted by your figure: it is generally recommended that you not use factors unless they are supported by at least 3 measured variables. But some of yours appear to have only 2 or 1. You may want to find more items that load primarily onto those factors to boost their reliability, or drop those items and factors. $\endgroup$
    – gung - Reinstate Monica
    Jan 15, 2012 at 21:40

2 Answers 2

Reset to default
1
$\begingroup$

i'm assuming that the purpose of your analysis is to obtain evidence for the validity of your scale/instrument. so, first of all, your instrument was designed based on 4 hypothesized constructs, therefore, you should approach this using confirmatory factor analysis (CFA). exploratory factor analysis (EFA) is appropriate when there is no a priori theory describing the relationship between observed variables (i.e., items) and constructs and can result in uninterpretable factors, as you see here.

then examine the results of your CFA model. the various fit statistics (e.g., X^2, RMSEA, modification indices, wald test statistics) can guide you through the refinement of your model.

if you prefer a more exploratory approach, also consider "backward search": Chou C-P, Bentler, P.M. (2002). Model modification in structural equation modeling by imposing constraints, Computational Statistics & Data Analysis, 41, (2), 271-287.

Improve this answer
$\endgroup$
4
$\begingroup$

A tough situation. Factors 6, 4, and 7 seem fairly robustly measured, but not the others, and I bet internal consistency is going to be low for factors 1, 3, and 5. Is it at all possible to assess reliability through some other method, such as interrater rel.? Or to assess validity through some other method than construct validity via factor analysis? Even if different scales (or individual items) get validated in different ways--sometimes you need to take whatever you can.

At any rate, I could see using v6 and v17 individually. Why force them into some multi-item scale when the loadings and correlations look like this.

And even given what I said above about coverage implying validity, I agree that you want to keep your eventual regression predictors pretty much unidimensional--especially since you have a large number of predictors, as with multidimensional variables the waters will get very, very muddy. This is particularly relevant since you seem to be adopting much more of an explanatory than a purely predictive mode (you care about causality).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yeah i care about the causality of the construct/item relations, because there is also the formative approach, which basically says "we got theese 20 items and they form the construct" as opposed to "the construct exists and it must be reflected by each individual items". The formative approach would get rid of they necessity of having high item correlations and unidimensionality.But the interpretation would be difficult. Still, thank you very much! $\endgroup$
    – Jack Shade
    Jan 16, 2012 at 13:16
  • $\begingroup$ What would a more predictive approach look like? $\endgroup$
    – Jack Shade
    Jan 16, 2012 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.