Someone in my organization just sent me a scale they developed (modified items from five separate scales used in published research) to measure an employee's readiness for change. They tested the internal consistency of the 15-item scale with Cronbach's alpha. The result was .995, and the alpha after each item was deleted was .995 (for each item). I'm pretty sure that results should not be this high and that there is a problem with the items themselves, the response scale, or perhaps the method of data collection. On the face of it, the individual items don't appear to be redundant. I know that a scale should also be tested for test-retest reliability, convergent and discriminant validity. What would be the first thing I should check to figure out what might be causing these results?
-
$\begingroup$ This is indeed a bit high even for an "individual testing" questionnaire. Do the usual alpha (based on covariances) and standardized alpha (based on correlations) differ much in your example? $\endgroup$– ttnphnsCommented Feb 10, 2015 at 7:22
-
$\begingroup$ Have you considered using IRT (en.wikipedia.org/wiki/Item_response_theory)? $\endgroup$– TimCommented Feb 10, 2015 at 7:36
-
$\begingroup$ Hi Tim, he provided a correlation matrix and several items are highly correlated >.97 and a few at .99. I don't have any information about the difference between the usual and standardized alpha's. He's also used a subset of the scale as the DV and regressed that on the other items. Not surprisingly the R square is over .90 for 5 of the items (.96 and .97 on two). Here I suspect that common method bias is a big problem. I don't and won't have access to the raw data to test the scale any further. I'm just leaning towards not using it based on the results he's provided. Thanks for your feedback. $\endgroup$– user3594490Commented Feb 10, 2015 at 16:54
-
1$\begingroup$ It would be great to see some examples. That high an alpha and that high interitem correlations suggest that the items are measuring the exact same construct. It appears that you only need to ask one of the items (any item). Are the items asking the same question but with slightly different format? $\endgroup$– doug.numbersCommented Mar 1, 2015 at 15:29
-
1$\begingroup$ No, at face value the question items were different. They were pulled from different scales (multiple dimensions of the construct). He selected one or two items from each of the other scales and combined into a single index (trying to create a unidiminsional scale). Instead of the alpha being low (items should be representing multiple dimensions), it was essentially 1. This should indicate that something else is inflating the alpha, no? I assumed the result was due to CMB, social desireability, or something else related to his method of data collection, participant characteristics, etc. $\endgroup$– user3594490Commented Mar 12, 2015 at 21:30
2 Answers
When I have ran into this kind of error, I had made a mistake during the coding process (I'm using SPSS). I usually use numbers like 99 or 999 to mark missing values. Once I forgot to specify the missing values to be excluded at the variable view, and I begun the scale analysis. I got very high C-alphas, around 0.95, just like you.
-
2$\begingroup$ To me this is a more plausible explanation than the otherwise good ideas about CMB, SD, etc. $\endgroup$– rolando2Commented Apr 10, 2016 at 11:45
I agree with @rolando2, as I believe mistakes made during the coding process could be the cause of such a high $\alpha$. Though my guess is that it is due to item redundancy - something you mentioned in your question. Another plausible cause could be your sample. I have analyzed survey data from many sources, and from my experience, it is clear that responses to organizational surveys are often quite lazy, in that it is common to see many respondents answer the same way across the entire survey. This is why I prefer organizational surveys that are both short and easy to read, as doing both makes it less likely for respondents to answer lazily.
