I'm working on a questionnaire that is intended to measure two constructs. There are presently 26 items in the questionnaire -- eleven for construct 1 and 15 for construct 2 -- and each item has five available responses: "strongly disagree" "disagree" "neutral" "agree" "strongly agree".

In our first iteration of the questionnaire, we used ten items for each construct, and had about 60 respondents. After the results came in, we made some judgments based on item discrimination (dichotomized), point-biserial correlation, cronbach's alpha after item deletion, and factor analysis, and dropped several of these questions and replaced them with others.

We now have 68 responses to the present questionnaire (the one with 26 items). I'd like to make sure I cover my bases as we do the reliability and validity analysis, and am specifically wondering if we should carry out additional diagnoses or approach the ones we've already calculated in different ways. Our approach to identifying potentially problematic items is to look for:

  1. item discriminations below .3 on the dichotomized responses - where agree/strongly is coded as 1 and the others are coded as zero, and we take the difference between the top 3'rd of respondents and the bottom 3'rd of respondents.

  2. low point-biserial (spearman's) correlations between items and the total score for the associated construct (total score = mean response for that construct). We don't have a threshold defining "low" ... and is spearman's or pearson's the appropriate measure?

  3. items whose inclusion causes cronbach's alpha to decrease -- identified by running tests with "cronbach's alpha after item is deleted." Would G6 or Omega be more appropriate here?

  4. Factor analysis -- we've been looking for items with low loadings in either dimension. Here, we're wondering about two thngs: a) whether to "swap" factors from their intended construct to the one which they appear to load with more strongly. Is that a defensible approach, even if the measure doesn't have face validity with respect to its intended construct? b) if an item loads strongly in both dimensions (where it appears that there's little separation) is that a problem? I didn't think it was but I thought it might be worthwhile to add.

If anyone has any thoughts on these questions, or if an important procedure comes to mind that we haven't consider, I'd love to hear it. Thanks in advance, and I apologize if this post is not properly directed!

  • $\begingroup$ Thanks to @Rolando2 for his insightful comments. It seems that I should definitely check out Corrected Item-Total Correlation; we were using spearman's ITC, which was each item's correlation with the sum of all (and not of all other) items. I am still wondering if using cronbach's alpha in the way that we mentioned - as a test of internal consistency, by comparing the statistic with and without the relevant item - is an appropriate way to judge these questions. I've heard here and there of G6 and omega but couldn't turn up much. $\endgroup$
    – dubhousing
    Apr 22 '12 at 2:07

Your method #1 loses information by dichotomizing in two different ways. I'd instead look at each item's correlation with the sum of all the other items (in software such as SPSS this is called "Corrected Item-Total Correlation"). For #2, where you have done something close to this, you could make a case for either Spearman's or Pearson's, and they'll hardly differ since with a 1-5 per-item range there shouldn't be many extreme outliers. You'll have to establish your own threshold, I'm afraid: how exacting do you want to be? How desirable is it to preserve a large number of items for your scales? And how concerned are you about your case-to-item ratio?

As for your questions about factor analysis, yes, to build the scales based on empirical criteria can be defensible, just as it can be to do so using a priori ideas about which item belongs to which dimension. Good research will hopefully reconcile any conflicts between the two. Items with multiple high loadings are a problem if you want uncorrelated factors, something that is often unrealistic in opinion research. At a more general level, I think you have some sense that factor analysis and scale development is best seen as a largely creative process where there are many subjective decisions to be made and often much work to do in justifying them!

  • $\begingroup$ Thanks Roland. Will try the corrected ITC, and appreciate your perspective on factor analysis. In this case probably won't be able to avoid correlation between factors. Couple questions - I'm not sure I follow your note about dichotomizing in two ways. The one that I recognize is taking a 5 point scale and making it 0/1 - what is the other? And, any insights into using cronbach's alpha to measure internal consistency in this context would be most welcome. Thanks again! $\endgroup$
    – dubhousing
    Apr 21 '12 at 23:38
  • $\begingroup$ You're welome @d. The other dichotomization was comparing top and bottom thirds. Not sure what your question is about alpha, though. What makes you doubt that it's an appropriate technique? $\endgroup$
    – rolando2
    Apr 22 '12 at 16:07
  • 1
    $\begingroup$ I've read in other areas that cronbach's alpha may not be the best measure of internal consistency for our kind of questionnaire, but i'm afraid i'm out of my depth when trying to understand exactly how to evaluate the alternatives. The ones I've seen cited include guttman's lambda 6, which is calculated by the function alpha in r package 'psych'; Revelle and Zinbarg discuss "Coefficients alpha, beta, omega and the glb" in their article of the same name (2008) and seem to hold that cronbach's alpha is a poor measure of reliability. $\endgroup$
    – dubhousing
    Apr 22 '12 at 20:54
  • $\begingroup$ Interesting. This is the first article I've seen that questions Cronbach's alpha's effectiveness when one wants to measure internal consistency or homogeneity. It is such a familiar technique, though, that if you replace it with another you'll probably have an uphill climb explaining to your audience why you're doing so. $\endgroup$
    – rolando2
    Apr 24 '12 at 2:25
  • $\begingroup$ Good point Rolando2. I will stick to the standard procedures but keep this question in mind for later. Thanks for your help. $\endgroup$
    – dubhousing
    Apr 24 '12 at 3:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.