# Combining items with low correlation and low internal validity using combinatory logic

So I want to combine four items into an index on strong theoretical grounds (scroll all the way down in case you are interested in what those grounds are) but looking at the statistics, I get abysmal results.

Cronbach's Alpha = 0.36


Correlation Matrix

                  Democracy   IndividualRights WomenRights   Veiling
Democracy         1.00000000  0.334327909     0.03009099 -0.042595581
IndividualRights  0.33432791  1.000000000     0.06226588 -0.008456227
WomenRights       0.03009099  0.062265877     1.00000000  0.370303230
Veiling          -0.04259558 -0.008456227     0.37030323  1.000000000


Conducting a multilevel analysis (N=14702, 12 Countries) with and without the index yielded the following standardized beta (with the dependent variable "Support for Islamism"). Standard errors are in parenthesis. Other coefficients have been left out for the sake of brevity.

Democracy                     -0.05 *** (0.01)
IndividualRights              -0.10 *** (0.01)
WomenRights                   -0.10 *** (0.01)
Veiling                       -0.02 *** (0.01)


Compared to this result if all variables are combined through addition:

LiberalInterpretation         -0.25 *** (0.01)


Question: So one can see that each effect is a strongly significant predictor, however when combining them the effect actually adds up and becomes even more important. So my intuition would be that this would be important to retain, a statistical approach seems to disagree though.

How to reconcile this?

I have a found a paper by Welzel and Inglehart, that seem to tap into what I am doing here.

"In combinatory logic, divergent variance among constituent components is not considered as measurement error but as complementary reality coverage. The methods literature characterizes combinatory constructs as “formative” and juxtaposes them to the dimensional logic of “reflective” constructs (Coltman et al., 2008). [...] The quality criteria for combinatory constructs are twofold. Theoretically, the combination must make sense such that the components meaningfully complement each other under an overarching idea. Empirically, the combination must make a difference in that it maps closer on its expected antecedents or consequences than does each of its components. Consequently, the yardstick to judge a combinatory construct is its predictedness and predictiveness relative to other aspects of reality. Datler, Jagodzinski, and Schmitt (2013) call this criterion external validity, in contrast to internal consistency."

You can find the paper here, citation is on page 7-8 https://www.researchgate.net/profile/Christian_Welzel2/publication/293636204_Misconceptions_of_Measurement_Equivalence_Time_for_a_Paradigm_Shift/links/57c5351608aecd4514159a62.pdf

So under this aspect, my index does not have internal validity but it does have strong theoretical reasons on which to combine them AND it has much greater predictiveness than each of its components alone (and thus EXTERNAL validity).

Under these circumstances, am I justified to combine those items? What are your opinions?

I am using RStudio, in case this is of any importance.

Background: I am trying to predict preference for Islamist politics in Arab countries. The four variables I want to combine are all items where respondents were asked if they agree with liberal interpretations of Islam (in regards of women's rights, mandatory veiling, individual rights and democracy). So they don't really correlate well since a lot of people agree on individual rights but not many do when it comes to women's rights, for example. So they naturally do not load on the same dimensions, but theoretically it makes a lot of sense that endorsing individual rights and gender equality as compatible with Islam would be part of a construct which I would call "Liberal Interpretations of Islam".

• I think in this case it would be because I am asking whether is is ok to combine the items on theoretical grounds.. so I think it is good to understand what these theoretical grounds are. Maybe I can put it at the end of the post, so the question is more central. Thanks for the suggestion.
– FaV1
Mar 19, 2017 at 14:19