Advice on scientifically sound scale construction

Question

I've been given a set of 20 Likert-items (ranging from 1-5, sample size n = 299) within the field of organizational research. The items are intended to measure a latent concept which is multidimensional, multifaceted and heterogenous in it's very nature. The goal is to create a scale(s) which can be nicely used to analyse different organizations and be used in logistic regression. Following the american association of psychology, a scale should be (1) unidimensional, (2) reliable and (3) valid.

Therefore we decided to select four dimensions or subscales with 4/6/6/4 items each; which are hypothesized to represent the concept.

The items werde constructed using the reflective approach (generating lots of possible items, and iteratively deleting items using cronbach's alpha and concept representation (validity) in three subsequent groups).

Using the available data, a preliminary parallel explanatory factor analysis based on polychoric correlations and using varimax rotation revealed that items load on other factors than expected . There are at least 7 latent factors as opposed to four hypothesized ones. The average inter-item correlation is quite low (r=0.15) albeit positive. The cronbach-alpha coefficient is also very low (0.4-0.5) for each scale. I doubt that an Confirmatory factor analysis would yield a good model fit.

If two dimensions were dropped, the cronbachs alpha would be acceptable (0.76,0.7 with 10 items per scale, which still could be made bigger by using the ordinal version of cronbachs alpha) but the scales themselves would still be multidimensional!

As I'm new to statistics and lack the appropriate knowledge, I am at a loss on how to proceed further. As I am reluctant to discard the scale(s) completely and resign to a descriptive-only approach, I've got different questions:

I) Is it wrong to use scales which are reliable, valid but not unidimensional?

II) Would it be appropriate to interpret the concept afterwards as formative and use the vanishing tetrad test to assess model-specification and use partial least squares (PLS) to arrive at a possible solution? After all, the concept seems to be more a formative than a reflective one.

III) Would using the item response models (Rasch, GRM etc) be of any use? As I've read, the rasch-models etc. also need the assumption unidimensionality

IV) Would it be appropriate to use the 7 factors as new "subscales"? Just discard the old definition and use a new one based on factor loadings?

I'd appreciate any thoughts on this one :)

EDIT: Added factor loadings & correlations

> fa.res$fa
Factor Analysis using method =  ml
Call: fa.poly(x = fl.omit, nfactors = 7, rotate = "oblimin", fm = "ml") 

Factor loadings calculated from factor pattern matrix and factor intercorrelation matrix, only values above 0.2 are displayed

Answer 1

i'm assuming that the purpose of your analysis is to obtain evidence for the validity of your scale/instrument. so, first of all, your instrument was designed based on 4 hypothesized constructs, therefore, you should approach this using confirmatory factor analysis (CFA). exploratory factor analysis (EFA) is appropriate when there is no a priori theory describing the relationship between observed variables (i.e., items) and constructs and can result in uninterpretable factors, as you see here.

then examine the results of your CFA model. the various fit statistics (e.g., X^2, RMSEA, modification indices, wald test statistics) can guide you through the refinement of your model.

if you prefer a more exploratory approach, also consider "backward search": Chou C-P, Bentler, P.M. (2002). Model modification in structural equation modeling by imposing constraints, Computational Statistics & Data Analysis, 41, (2), 271-287.

Answer 2

A tough situation. Factors 6, 4, and 7 seem fairly robustly measured, but not the others, and I bet internal consistency is going to be low for factors 1, 3, and 5. Is it at all possible to assess reliability through some other method, such as interrater rel.? Or to assess validity through some other method than construct validity via factor analysis? Even if different scales (or individual items) get validated in different ways--sometimes you need to take whatever you can.

At any rate, I could see using v6 and v17 individually. Why force them into some multi-item scale when the loadings and correlations look like this.

And even given what I said above about coverage implying validity, I agree that you want to keep your eventual regression predictors pretty much unidimensional--especially since you have a large number of predictors, as with multidimensional variables the waters will get very, very muddy. This is particularly relevant since you seem to be adopting much more of an explanatory than a purely predictive mode (you care about causality).