1
$\begingroup$

So I want to combine four items into an index on strong theoretical grounds (scroll all the way down in case you are interested in what those grounds are) but looking at the statistics, I get abysmal results.

Cronbach's Alpha = 0.36
Kaiser-Meyer-Olkin factor adequacy = 0.5

Correlation Matrix

                  Democracy   IndividualRights WomenRights   Veiling
Democracy         1.00000000  0.334327909     0.03009099 -0.042595581
IndividualRights  0.33432791  1.000000000     0.06226588 -0.008456227
WomenRights       0.03009099  0.062265877     1.00000000  0.370303230
Veiling          -0.04259558 -0.008456227     0.37030323  1.000000000

Conducting a multilevel analysis (N=14702, 12 Countries) with and without the index yielded the following standardized beta (with the dependent variable "Support for Islamism"). Standard errors are in parenthesis. Other coefficients have been left out for the sake of brevity.

Democracy                     -0.05 *** (0.01)
IndividualRights              -0.10 *** (0.01)
WomenRights                   -0.10 *** (0.01)
Veiling                       -0.02 *** (0.01)

Compared to this result if all variables are combined through addition:

LiberalInterpretation         -0.25 *** (0.01)

Question: So one can see that each effect is a strongly significant predictor, however when combining them the effect actually adds up and becomes even more important. So my intuition would be that this would be important to retain, a statistical approach seems to disagree though.

How to reconcile this?

I have a found a paper by Welzel and Inglehart, that seem to tap into what I am doing here.

"In combinatory logic, divergent variance among constituent components is not considered as measurement error but as complementary reality coverage. The methods literature characterizes combinatory constructs as “formative” and juxtaposes them to the dimensional logic of “reflective” constructs (Coltman et al., 2008). [...] The quality criteria for combinatory constructs are twofold. Theoretically, the combination must make sense such that the components meaningfully complement each other under an overarching idea. Empirically, the combination must make a difference in that it maps closer on its expected antecedents or consequences than does each of its components. Consequently, the yardstick to judge a combinatory construct is its predictedness and predictiveness relative to other aspects of reality. Datler, Jagodzinski, and Schmitt (2013) call this criterion external validity, in contrast to internal consistency."

You can find the paper here, citation is on page 7-8 https://www.researchgate.net/profile/Christian_Welzel2/publication/293636204_Misconceptions_of_Measurement_Equivalence_Time_for_a_Paradigm_Shift/links/57c5351608aecd4514159a62.pdf

So under this aspect, my index does not have internal validity but it does have strong theoretical reasons on which to combine them AND it has much greater predictiveness than each of its components alone (and thus EXTERNAL validity).

Under these circumstances, am I justified to combine those items? What are your opinions?

I am using RStudio, in case this is of any importance.

Background: I am trying to predict preference for Islamist politics in Arab countries. The four variables I want to combine are all items where respondents were asked if they agree with liberal interpretations of Islam (in regards of women's rights, mandatory veiling, individual rights and democracy). So they don't really correlate well since a lot of people agree on individual rights but not many do when it comes to women's rights, for example. So they naturally do not load on the same dimensions, but theoretically it makes a lot of sense that endorsing individual rights and gender equality as compatible with Islam would be part of a construct which I would call "Liberal Interpretations of Islam".

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ I think in this case it would be because I am asking whether is is ok to combine the items on theoretical grounds.. so I think it is good to understand what these theoretical grounds are. Maybe I can put it at the end of the post, so the question is more central. Thanks for the suggestion. $\endgroup$
    – FaV1
    Mar 19, 2017 at 14:19
  • $\begingroup$ This thread might be partly helpful to read. $\endgroup$
    – ttnphns
    Mar 19, 2017 at 14:41
  • $\begingroup$ Thanks for the link. So I am guessing you are referring to the part where you talk about calculating Euclidean or Manhattan distance for uncorrelated variables? It sounds interesting and I haven't heard about this at all before. Do you have some example of someone doing this to construct an index (preferably in R)? My background is Political Science, so this stuff can be difficult for me. A first google search didn't really yield relavant answers unfortunately. $\endgroup$
    – FaV1
    Mar 19, 2017 at 15:02
  • $\begingroup$ I did not refer to any specific part of that answer. The answer simply represented a a reasoning. Weakly correlated items are strange to sum (average) into a single index. It can be done sometimes under special motives but that won't be statistically warranded. Why do you resist the idea to keep all your important items as separate indicators? $\endgroup$
    – ttnphns
    Mar 19, 2017 at 15:09
  • $\begingroup$ The most important reasons are that my single index would be based on theory and that I subsequently wanted to calculate interaction effects with that index. I guess I could do those one by one, but this seems awfully complex, especially if a combination yields better predictive results and can be justified theoretically. I guess my ultimate question is, is it possible to justify the combination on theoretical grounds, ignore the internal validity and focus on the external validity (as Welzel and Inglehart suggest in their paper)? And if so, should I then use distance rather than summing? $\endgroup$
    – FaV1
    Mar 19, 2017 at 15:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.