2
$\begingroup$

I'm wondering exactly what the question says: Why does a factor (as linear combination of its items) need to be internally consistent?

If I want to use a factor found in a factor analysis as a dependent variable in some regression and the factor isn't enough internally consistent (e.g. Cronbach's alpha of items composing the factor is between .5 and .7), does that mean my estimate of beta (the regression coefficient) will have a larger standard error; or does it violate the assumptions of the regression more generally? Can I trust my coefficients and p-values in a regression if my factor, the scale, is not internally consistent?

I realize there are ways to make the factor more internally consistent by removing bad fitting items or adding more good items, but my question is about is really, and why, internal consistency is important in the first place for a factor.

$\endgroup$
  • 1
    $\begingroup$ Please get to read stats.stackexchange.com/a/287619/3277 and the whole thread there. Term "internal consistency" is dubious, I would object against using it altogether. There's something from reliability, something from validity. Alpha is one of homogeneity measures of reliability. Alpha can be high in quite unsatisfactory factor, and it can be enough low when items are few and factor is relatively convincing. $\endgroup$ – ttnphns Nov 30 '17 at 22:15
  • 1
    $\begingroup$ Factor does not have to be "internally consistent" (whatever that means, especially when it means high alpha). It has to explain well the observed correlations; and, when there is an external criterion of the trait, it must well correlate with it (and weaker correlate with other traits). $\endgroup$ – ttnphns Nov 30 '17 at 22:24
  • 1
    $\begingroup$ @Brett High (internal consistency) reliability is not a regression assumption. It's one of the things supposed to be needed for the measurement to be seen as meaningful. It's not a statistics issue but a scientific one (at least in psychometrics -- of which I have no expertise). It would be very unusual for a book on statistical theory to say anything about it (except, perhaps, as far as theory could inform the calculations that the scientific arguments are based on, so it's possible some books might discuss the calculations) $\endgroup$ – Glen_b Dec 1 '17 at 4:28
2
$\begingroup$

High item-item internal homogeneity (what Alpha is mostly about) is not necessary for the items to be able to represent a latent factor validly (in the sense: unbiasedly).

Here is three variables data, V1, V2, V3, 50 cases. In one dataset (first 3 columns), correlations between all the three are 0.8. In the other dataset (next three columns) correlations between all the three are 0.3. The variables are standardized.

    v1_0.8     v2_0.8     v3_0.8     v1_0.3     v2_0.3     v3_0.3    fsc_0.8    fsc_0.3

 -.0216322  -.3145886   .1604344   .0233404  -.5238589   .3650942  -.0604708  -.0463575
  .9333728  2.3119911  1.9546955  -.0931651  2.4597177  1.7832511  1.7888294  1.4205299
  .3052711  1.0415126  -.6526027   .3129150  1.6869127 -1.4835223   .2387995   .1767378
  .5030737  1.1074983   .0701299   .3364104  1.4587923  -.4845117   .5781644   .4486660
 1.2797719  1.1299271  1.4849064  1.0008518   .7009884  1.3591254  1.3397510  1.0478070
 2.2660289  2.7770475  2.0257540  1.7062525  2.6268757  1.2105131  2.4316898  1.8976580
 -.2145210  -.1405609  -.2963220  -.1686681  -.0270311  -.3174341  -.2240841  -.1756520
 -.5332263  -.4872407 -1.1308836  -.2352714  -.1383810 -1.3392088  -.7400683  -.5863339
 -.4490196  -.3887868 -1.2903304  -.0895620   .0339002 -1.6494411  -.7320827  -.5836781
 1.0338865   .6809572   .5711352  1.1060618   .4344277   .2254950   .7863807   .6045187
 -.4836489  -.9900844  -.6503464  -.1494410 -1.0862012  -.4473419  -.7306871  -.5761066
-1.9360488 -2.1891431 -1.6425752 -1.5526669 -1.9972980  -.9659407 -1.9841218 -1.5458512
-1.2570528 -1.8665392 -1.3973651  -.7377985 -1.8553405  -.9706582 -1.5552170 -1.2199325
 -.0546157  -.0376983  -.4180892   .0754412   .1096984  -.6011515  -.1755796  -.1424061
-1.3231616  -.2141172  -.9050470 -1.5870724   .4998681  -.7890407  -.8401643  -.6422622
  .0691481 -1.0107037 -1.7590763  1.0631309  -.9432335 -2.3390808  -.9290220  -.7596543
  .1556118   .3593251  -.1902539   .1709739   .5504975  -.4781568   .1116915   .0832896
 1.2228620  1.1568216  1.5301791   .8909594   .7477838  1.4402742  1.3449995  1.0539863
-2.5730333 -1.7875258 -1.4853883 -2.6979986 -1.1993673  -.6252286 -2.0110160 -1.5481409
 1.9842442  1.1221856  1.7050726  1.9785803   .3418057  1.4249476  1.6551651  1.2820747
  .9770592  -.0441422   .8725782  1.1729472  -.7465142   .9657704   .6210934   .4765687
-1.2069782  -.3847021  -.1605063 -1.6095356  -.0626192   .3594374  -.6027552  -.4493597
 -.7172861  -.8802134 -1.0844749  -.3874150  -.6787273 -1.0567414  -.9226038  -.7266897
 -.2682062   .3626536  -.1105669  -.4896474   .6906040  -.1947034  -.0055451   .0021405
 1.6929962  1.2566269  1.2078208  1.6677241   .8306217   .7329766  1.4301679  1.1061222
-1.4159937 -1.1830667  -.6379551 -1.4762627 -1.0244063   .0003235 -1.1135389  -.8558996
 -.1500775  -.6646840  -.0972387   .0435106  -.9146387   .1483573  -.3137296  -.2474135
 -.1828678 -1.1630316  -.5886179   .3388003 -1.4851117  -.4074865  -.6654773  -.5318845
  .6590666   .4445127   .6415163   .6060979   .1959781   .5618709   .6003159   .4668962
 1.5162195   .6415474  2.0699336  1.3243912  -.3331751  2.3326103  1.4543363  1.1377875
  .2245677  -.0234253   .9045059   .0324350  -.4371384  1.2971452   .3803449   .3054940
 -.0926322  -.0904877   .1399430  -.1543786  -.1501836   .2809690  -.0148529  -.0080762
  .3780963  -.2112872  -.1955229   .7054150  -.3969598  -.3673892  -.0098776  -.0201739
 -.4290316 -1.1542862  -.9926653   .1074007 -1.2363992  -.9300543  -.8861426  -.7048396
 -.7585939  -.2935150  -.0498380 -1.0109333  -.1354595   .3220680  -.3790716  -.2821767
 -.0843910   .1888064   .0036812  -.1935725   .3169627  -.0295486   .0371854   .0321232
  .8418637   .4953959  -.2870958  1.1798177   .5265572  -.9389054   .3612582   .2627144
 -.4998451   .4540655   .1387108  -.9529158   .8310741   .2409140   .0319685   .0407599
  .2479741  -.0448949   .8637955   .0888841  -.4644383  1.2339117   .3670067   .2938266
 -.7611045 -1.1167585 -1.5739342  -.2013707  -.8493105 -1.6992832 -1.1874242  -.9413473
  .0660042   .3296798   .5710234  -.2137543   .2746073   .7246142   .3325490   .2688753
-1.4107885 -1.7275688  -.9734133 -1.1616271 -1.7337159  -.3165378 -1.4144562 -1.0994671
 -.1167343   .4900621  -.7025693  -.1052139  1.0316952 -1.1989958  -.1132596  -.0932851
 -.2279270  -.4127442   .4400241  -.3458270  -.6906678   .9049981  -.0690230  -.0450129
  .1772902   .2251722   .1172732   .1448735   .2318547   .0292001   .1787899   .1389544
 1.2545414   .6914799   .5711250  1.4314428   .3655735   .1365945   .8659027   .6618993
-1.3840245   .0592204  -.1973831 -2.0171455   .6902884   .2124831  -.5236350  -.3814642
 1.0357746   .2783454   .9829308  1.1091084  -.3193657   .9952914   .7901894   .6110396
  .0793607  1.1721525   .8984689  -.6028225  1.4306523   .9152923   .7395975   .5966926
 -.3516433   .0448097  -.4555760  -.3837007   .3618054  -.5731659  -.2622702  -.2036969

Both the "0.8" set and the "0.3" set were generated from the same random normal values "ingot" by method described here; because of that, the two obtained bunches of variables-vectors, "0.8" bunch and "0.3" bunch, are oriented in the space of individuals (cases) the same way and the only difference between the two is the angles between the vectors: the "0.8" bunch is more tight vectors and the "0.3" bunch is more loose vectors.

Cronbach's alpha in set "0.8" is .923 and in set "0.3" is .563. It is clear that "internal consistency" (if to let Alpha personify that phrase) is much higher there where correlations are bigger then where correlations are smaller. Alpha is also dependent on the number of items in the construct, but here we have equal number of variables in both sets.

Let us assume that the construct uniting the three items for us is representing a latent factor, i.e. factor-based construct. Then factor analysis should support the construct. Perform factor analysis extracting one factor.

Loadings obtained in FA of "0.8" set
LOAD 
   .8944044699 
   .8944044699 
   .8944044699 

They restore correlations ideally 
LOAD*T(LOAD) [commulalities on the diagonal]
   .7999593558   .7999593558   .7999593558 
   .7999593558   .7999593558   .7999593558 
   .7999593558   .7999593558   .7999593558


Loadings obtained in FA of "0.3" set
LOAD 
   .5477000812 
   .5477000812 
   .5477000812 

They restore correlations ideally     
LOAD*T(LOAD) [commulalities on the diagonal]
   .2999753790   .2999753790   .2999753790 
   .2999753790   .2999753790   .2999753790 
   .2999753790   .2999753790   .2999753790

You see that in both cases FA extracted a factor which successfully explained the observed correlations (and thus the validity of the factor was found [I don't say it was confirmed, since we do EFA, not confirmatory FA nor cross-validation]). However, loadings (which are the correlations of a factor with items) were higher in "0.8" data than in "0.3" data. In other words, items in "0.3" set are only weakly driven by the common factor; they are still considerably individualized ("unique") and that's why correlations among them are as low as 0.3.

Let's compute factor scores of the factor in both analyses (see data columns fsc_0.8 and fsc_0.3 above). Here we used Regression method, but other methods, such as Anderson-Rubin, might be used as well. Because both data sets were generated as proportionally identical (except the strength of the correlations) in the space of individuals, factor scores from the two factor analyses correlate almost perfectly. In other words, we have yet another evidence of having the same latent factor operating in set "0.8" and set "0.3"; the only difference being between the sets that the factor loads items strongly in "0.8" and weaker in "0.3". So, variables in both our sets are equally valid to represent the factor in the sense that they belong to the same factor without any bias, but in one set they belong to the factor much and in the other set they belong to the factor little. Since factor scores fsc_0.8 and fsc_0.3 are linearly identical, it's no difference which of them you input to, say, a regression analysis as a dependent variable.

Of course, items which belong together much are closer to be duplicates of each other and will have higher Alpha internal tightness. But, as we see, low alpha is still compatible with good validity in the sense of factor inbiasedness. Validity is not the same as reliability (1, 2).

Moreover, if we turn to speak away from construct validity towards external validity, i.e. the correlation of the factor (e.g. of factor scores or of sum of items) with some outside criterion variable embodying the trait, we might find that it is not necessarily so that the correlation will be higher in set "0.8" than in set "0.3".

Why would we, despite all said, prefer to have data "0.8" than "0.3"? Because if we add random noise (i.e. add measurement error) to the variables, correlations in "0.8" will drop to, say, 0.6, which is enough eligible for a factor analysis to show good results in real settings, whereas "0.3" will drop to 0.1 which is already too low correlations to extract the factor mathematically consistent (in real data settings). It is more difficult, having the same detection apparatus, to find a small needle in a vast haystack than an awl in a tight haystack.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Brilliant, thank you for the explanation. This is so clear with simulated data! Wish I could upvote you more than once. $\endgroup$ – Brett Dec 1 '17 at 13:25
1
$\begingroup$

@ttnphns @Glen_b thanks for your answers and for the link to the reliability/validity discussion. I will have to read it a few more times to fully understand it. For the time being, what I take from your answers is that a low cronbach's alpha is not in itself a cause for concern. Reliability between items in a factor is purely a theoretical concern and does not jeopardize my regression results. If my factor model and regression model fit reasonably well and I have meaningful, theory-driven factors and interesting regression results, then I should feel free to report those results.

Please correct me if I'm wrong!

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.