what to do with unreliable Likert scales

Question

I have a 20 question 5 point Likert scale questionairre built to tap four constructs, each via 5-items. Was hoping to use PCA to reduce data and then use parametric tests for each construct.

In retrospect prob naive.

Data are markedly non-normal - very heavily skewed to Agree side for most items. Cronbach Alpha Reliability for each "construct" is about 0.45-0.5 No correlation coefficients over 0.3 in correlation matrix Reflect & Log Transformation of variables doesn't help much. So, I think I need to abandon parametrics.

Is it reasonable to do the following: Note the above and then:

present a table cof all individual items grouped according to "construct" each with the median likert score value and range.

Or,

For each 5 item "scale" add the scores for each of the points over the 5 items (so add all the agrees, all the strongly agrees etc) Present one frequency histogram of the overall "scale" and just refer to the median and range for that scale?

If not, can anyone suggest the best way to refer to those data in an appropriately qualified descriptive non-parametric way? These are not central to findings: supplementary to strong qualitative data. many thanks in anticipation.

Answer 1

A few points to consider

1. Generally, a 5-point Likert-scale data provides a poor approximation of continuous measurement in the first place. So note that calculating the normality of individual ordinal variables is usually a dismal and questionable practice. Having said that, for each of your subconstructs you essentially calculate scale averages, so each of your subconstruct scores may have some degree of continuous measurement. So the question: how badly skewed is each of your subconstructs? If skew and kurtosis are within |2.00| (Gravetter, & Wallnau, 2014; Trochim & Donnelly, 2006) you may be just fine.

2. Your Cronbach Alpha is indeed very low. So you may try a number of remedies, including a) calculate Alpha for the entire 20-item construct, even though you might not have planned it in advance. If your Alpha is at least > .60 you are fine. If it is above .07 then it is good; b) for each construct, inspect correlations among their respective 5 items. If you see that one of the items has a close to zero and (possibly non-significant) correlation with other items, you may try to recalculate Alpha without that item, and your Alpha may increase to acceptable levels. Indeed, in scale development, some authors report the Alpha coefficient when one item is deleted. Of course, it is understandable that you only have 5 items and deleting any of those may not be an option to consider. Further, if you are interested, you may read this similar thread, in which the OP had low levels of Alpha, and factors affecting it were succinctly summarised with appropriate references.

   If your endeavor with the Alpha fails, you may consider calculating McDonald's Omega reliability for each subconstruct. Essentially, Omega is a much more robust measure of reliability than Alpha because it is based on far more realistic assumptions. For example, each item does not need to load with the same magnitude on each of your (latent) subconstructs. If you are familiar with R, then calculating Omega is essentially a one-liner psych::omega() Just make sure you have installed and loaded the psych package.

3. Inter-scale correlations in the region of .30 are fine and are commonly found in social sciences. So reporting those should not be a problem. Unless correlations of such magnitude are explicitly thought to be low in your field.

4. Again if you are familiar with R or Mplus, testing your subconstructs with Confirmatory Factor Analysis might give you useful insights, as to how well your models fit. Or if they do not, modification indices might signal sources of misfit.

---

References

Gravetter, F., & Wallnau, L. (2014). Essentials of statistics for the behavioral sciences. Belmont, CA: Wadsworth.

Trochim, W. M., & Donnelly, J. P. (2006). The research methods knowledge base (3rd ed.). Cincinnati, OH: Atomic Dog.