4
$\begingroup$

I am doing a Masters project (in environmental sociology) which includes a series of 15 questions that use a 6-point Likert scale to respond to various statements. In my pilot survey, my supervisor calculated Cronbach's alpha and said the internal consistency is too low (alpha = 0.63) and suggested I redo the questions.

Personally, I am confused about the need to assess the internal consistency in my research. The Likert Scales in my research assess a range of statements across different themes so the responses are not measuring/validating the same construct. If anything, the statements are designed to assess the potential variation in perceptions across those different themes.

My question is simple. If I use a Likert Scale, does that also require a high internal consistency even if the statements involve different themes?

$\endgroup$
2
  • $\begingroup$ Will you be combining any of your 15 questions into (an) overall score(s)? $\endgroup$
    – awhug
    Sep 20, 2022 at 9:55
  • $\begingroup$ @awhug - no I am not combining any of the questions. Each one is treated separately. $\endgroup$
    – anna6931
    Sep 20, 2022 at 10:00

2 Answers 2

3
$\begingroup$

Not necessarily, no. @Student100 gives good answer but misses one important point, measures of internal consistence assume that the measurements are theeffect of the construct, that is, are they caused by the construct. For example:

  • Whether you get a math question correct is caused by your math ability.
  • How you answer a question about having high self esteem is caused by your self esteem.

These are called effect indicators, because the indicator is the effect of the construct.

But your construct might have causal indicators. A causal indicator is the cause of the construct. And it's often not clear:

  • I don't have many friends.
  • Sometimes I feel alone.

Are these the effect of loneliness (in which case alpha would be appropriate) or the cause of loneliness, in which case it is not appropriate.

A more extreme example is a measure of stress, which might look at events that happened to you recently, and how much they affected you:

  • Lost job
  • Death in family
  • Moved house
  • Been ill

There is not a construct of stress that is the cause of each of these things. Rather each of these things adds stress, and the more there are, the greater the impact.

The classic reference on this is Bollen and Lennox (1991): https://www.researchgate.net/publication/232516427_Conventional_Wisdom_on_Measurement_A_Structural_Equation_Perspective - but there has been lots of work since.

$\endgroup$
0
$\begingroup$

Great question. As I am sure you know internal consistency is the extent to which items on an assessment/test/scales/survey measure the same thing. In a vacuum (i.e., irrespective of the item content), a higher value of Cronbach's $\alpha$ is preferable to a smaller one. This notion is reflected by the existence of various rules of thumb that are often used to indicate whether a particular value of Cronbach's $\alpha$ (or any other reliability measure for that matter). Common rules of thumb define values 0.9 $\ge$ as Excellent, values between 0.9 and 0.8 as Good, values between 0.8 and 0.7 as Acceptable, values between 0.7 and 0.6 as Questionable, values between 0.6 and 0.5 as Poor, and values < 0.5 as Unacceptable. I am not sure if your advisor was using similar rules of thumb to assess your survey, but is possible.

Now to your question.

If I use a Likert Scale, does that also require a high internal consistency even if the statements involve different themes?

You are correct in asking this, as scales with higher values of $\alpha$ are not always ideal, as it would indicate your construct is very narrow, and as a result, maybe not of much substantive interest. This is not to say your scale does not need to be reassessed (which can be done by removing items, adding items, or, as your advisor suggested - rewording items); it is just to say that how broad your construct is intended to be matters when choosing what value of $\alpha$ is acceptable for your application.

It should also be added that the reliability (regardless of how it is estimated, e.g., via $\alpha$, or latent variable models such as factor analysis and item response theory) of a construct (e.g., something related to environmental sociology) using the same set of items (i.e., your 15 items) will likely be different when administered to different populations. This should also be considered when evaluating estimates of $\alpha$. For instance, are you administering this survey to a sample of college students in majors unrelated to the environment? If this were the case, I could imagine lower sample estimates of $\alpha$ being observed when compared to a sample of students who are in a more related major (e.g., environmental studies). Because such students would likely have a less cohesive (i.e., less reliable) internal representation of environmental issues.

Finally, because your survey contains ordinal responses, I suggest you consider estimating internal consistency using Zumbo's ordinal $\alpha$$^1$ as it was designed for instances such as yours, where classical test theory (CTT) methods are used to assess reliability. See Gadermann, Guhn, & Zumbo (2012) and Zumbo, Gadermann, & Zeisser (2007) for more information on ordinal $\alpha$.

$^1$ Note that ordinal $\alpha$ is not without controversy. Recently, Chalmers (2018) made practical and theoretical arguments against the use of ordinal $\alpha$, positing that it is only useful as a theoretical exercise. However, like most psychometric arguments, there is strong disagreement (Zumbo and Kroc, 2019). Personally, I would use ordinal $\alpha$, though you should be fine using either.

References

Chalmers, R. P. (2018). On misconceptions and the limited usefulness of ordinal alpha. Educational and Psychological Measurement, 78(6), 1056-1071.

Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, and Evaluation, 17(1), 3.

Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of modern applied statistical methods, 6(1), 4.

Zumbo, B. D., & Kroc, E. (2019). A measurement is a choice and Stevens’ scales of measurement do not help make it: A response to Chalmers. Educational and Psychological Measurement, 79(6), 1184-1197.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.