Assessing reliability of a questionnaire: dimensionality, problematic items, and whether to use alpha, lambda6 or some other index?

Question

I am analyzing scores given by participants attending an experiment. I want to estimate the reliability of my questionnaire which is composed of 6 items aimed at estimating the attitude of the participants towards a product.

I computed Cronbach's alpha treating all items as a single scale (alpha was about 0.6) and deleting one item at a time (max alpha was about 0.72). I know that alpha can be underestimated and overestimated depending on the number of items and the dimensionality of the underlying construct. So I also performed a PCA. This analysis revealed that there were three principal components explaining about 80% of the variance. So, my questions are all about how can I proceed now?

• Do I need to perform alpha computation on each of these dimension?
• Do I have remove the items affecting reliability?

Further, searching on the web I found there is another measure of reliability: the lambda6 of guttman.

• What are the main differences between this measure and alpha?
• What is a good value of lambda?

Answer 1

I think @Jeromy already said the essential so I shall concentrate on measures of reliability.

The Cronbach's alpha is a sample-dependent index used to ascertain a lower-bound of the reliability of an instrument. It is no more than an indicator of variance shared by all items considered in the computation of a scale score. Therefore, it should not be confused with an absolute measure of reliability, nor does it apply to a multidimensional instrument as a whole. In effect, the following assumptions are made: (a) no residual correlations, (b) items have identical loadings, and (c) the scale is unidimensional. This means that the sole case where alpha will be essentially the same as reliability is the case of uniformly high factor loadings, no error covariances, and unidimensional instrument (1). As its precision depends on the standard error of items intercorrelations it depends on the spread of item correlations, which means that alpha will reflect this range of correlations regardless of the source or sources of this particular range (e.g., measurement error or multidimensionality). This point is largely discussed in (2). It is worth noting that when alpha is 0.70, a widely refered reliability threshold for group comparison purpose (3,4), the standard error of measurement will be over half (0.55) a standard deviation. Moreover, Cronbach alpha is a measure of internal consistency, it is not a measure of unidimensionality and can’t be used to infer unidimensionality (5). Finally, we can quote L.J. Cronbach himself,

Coefficients are a crude device that does not bring to the surface many subtleties implied by variance components. In particular, the interpretations being made in current assessments are best evaluated through use of a standard error of measurement. --- Cronbach & Shavelson, (6)

There are many other pitfalls that were largely discussed in several papers in the last 10 years (e.g., 7-10).

Guttman (1945) proposed a series of 6 so-called lambda indices to assess a similar lower bound for reliability, and Guttman's $\lambda_3$ lowest bound is strictly equivalent to Cronbach's alpha. If instead of estimating the true variance of each item as the average covariance between items we consider the amount of variance in each item that can be accounted for by the linear regression of all other items (aka, the squared multiple correlation), we get the $\lambda_6$ estimate, which might be computed for multi-scale instrument as well. More details can be found in William Revelle's forthcoming textbook, An introduction to psychometric theory with applications in R (chapter 7). (He is also the author of the psych R package.) You might be interested in reading section 7.2.5 and 7.3, in particular, as it gives an overview of alternative measures, like McDonald's $ \omega_t$ or $\omega_h$ (instead of using the squared multiple correlation, we use item uniqueness as determined from an FA model) or Revelle's $\beta$ (replace FA with hierarchical cluster analysis, for a more general discussion see (12,13)), and provide simulation-based comparison of all indices.

References

1. Raykov, T. (1997). Scale reliability, Cronbach’s coefficient alpha, and violations of essential tau-equivalence for fixed congeneric components. Multivariate Behavioral Research, 32, 329-354.
2. Cortina, J.M. (1993). What Is Coefficient Alpha? An Examination of Theory and Applications. Journal of Applied Psychology, 78(1), 98-104.
3. Nunnally, J.C. and Bernstein, I.H. (1994). Psychometric Theory. McGraw-Hill Series in Psychology, Third edition.
4. De Vaus, D. (2002). Analyzing social science data. London: Sage Publications.
5. Danes, J.E. and Mann, O.K.. (1984). Unidimensional measurement and structural equation models with latent variables. Journal of Business Research, 12, 337-352.
6. Cronbach, L.J. and Shavelson, R.J. (2004). My current thoughts on coefficient alpha and successorprocedures. Educational and Psychological Measurement, 64(3), 391-418.
7. Schmitt, N. (1996). Uses and Abuses of Coefficient Alpha. Psychological Assessment, 8(4), 350-353.
8. Iacobucci, D. and Duhachek, A. (2003). Advancing Alpha: Measuring Reliability With Confidence. Journal of Consumer Psychology, 13(4), 478-487.
9. Shevlin, M., Miles, J.N.V., Davies, M.N.O., and Walker, S. (2000). Coefficient alpha: a useful indicator of reliability? Personality and Individual Differences, 28, 229-237.
10. Fong, D.Y.T., Ho, S.Y., and Lam, T.H. (2010). Evaluation of internal reliability in the presence of inconsistent responses. Health and Quality of Life Outcomes, 8, 27.
11. Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255-282.
12. Zinbarg, R.E., Revelle, W., Yovel, I., and Li, W. (2005). Cronbach's $\alpha$, Revelle's $\beta$, and McDonald's $\omega_h$: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123-133.
13. Revelle, W. and Zinbarg, R.E. (2009) Coefficients alpha, beta, omega and the glb: comments on Sijtsma. Psychometrika, 74(1), 145-154

Answer 2

Here are some general comments:

• PCA: The PCA analysis does not "reveal that there are three principal components". You chose to extract three dimensions, or you relied on some default rule of thumb (typically eigenvalues over 1) to decide how many dimensions to extract. In addition eigenvalues over one often extracts more dimensions than is useful.
• Assessing item dimensionality: I agree that you can use PCA to assess the dimensionality of the items. However, I find that looking at the scree plot can provide a better guidance for number of dimensions. You may want to check out this page by William Revelle on assessing scale dimensionality.
• How to proceed?
  • If the scale is well established, then you may want to leave it as is (assuming its properties are at least reasonable; although in your case 0.6 is relatively poor by most standards).
  • If the scale is not well established, then you should consider theoretically what the items are intended to measure and for what purpose you want to use the resulting scale. Given that you have only six items, you do not have much room to create multiple scales without dropping to worrying numbers of items per scale. Simultaneously, it is a smart idea to check whether there are any problematic items either based on floor, ceiling, or low reliability issues. Also, you may want to check whether any items need to be reversed.
  • I put together some links to general resources on scale development that you may find helpful

The following addresses your specific questions:

• Do I need to perform alpha computation on each of these dimension?
  • As you may gather from the above discussion, I don't think you should treat your data as if you have three dimensions. There are a range of arguments that you could make depending on your purposes and the details, so it's hard to say exactly what to do. In most cases, I'd be looking to create at least one good scale (perhaps deleting an item) rather than three unreliable scales.
• Do I have remove the items affecting reliability?
  • It's up to you. If the scale is established, then you may choose not to. If your sample size is small, it might be an anomaly of random sampling. However, in general I'd be inclined to delete an item if it was really dropping your alpha from 0.72 to 0.60. I'd also check whether this problematic item isn't actually meant to be reversed.

I'll leave the discussion of lambda 6 (discussed by William Revelle here) to others.