2
$\begingroup$

I have used a questionnaire to collect data from 200 people.

This questionnaire was obtained from a published research paper in which researchers had used it in a specific population. It involves 34 items that are nominal multiple choice questions with one possible answer.

For example: Which factor do you consider?

  1. X
  2. Y
  3. Z

Questions do not have all the same number of possible answers. For example, question1 has 3 possible answers, question2 4 and so on.

According to the paper, Cronbach's $\alpha$ was 0.62.

I wanted to use the questionnaire in a different population. I carried out a pilot study in 20 people and made some changes. My colleague in charge of the statistics calculated Cronbach's $\alpha$. It was 0.65. When I finished the study, gathering data from 200 people, the calculated Cronbach's $\alpha$ was 0.57.

However, I myself think that Cronbach's $\alpha$ is not suitable in this case, as my questionnaire is not Likert, nor scaling. I believe that neither was it suitable in the original published paper. Unfortunately, time has passed and test-retest cannot be performed.

So, do you think that Cronbach's $\alpha$ can be applied in this questionnaire? Which is the correct way of coding these data (different coding results in a different Cronbach's $\alpha$)? What other reliability tests can be performed in this case?

Improve this question
$\endgroup$
10
  • $\begingroup$ Is there one answer that is correct, and one that is not correct? How is the questionnaire scored? $\endgroup$
    – Jeremy Miles
    Dec 29, 2016 at 1:58
  • $\begingroup$ Each question has multiple possible answers, but people taking the survey have to choose only one. Questions investigate patients' personal opinion for something. $\endgroup$
    – NickPap
    Dec 29, 2016 at 5:15
  • 1
    $\begingroup$ And how is it scored? $\endgroup$
    – Jeremy Miles
    Dec 29, 2016 at 15:09
  • 1
    $\begingroup$ If it's not scored, there is no meaning to reliability, except the reliability of individual items (which you could assess with retesting). But don't you then end up with 34 variables? What do you do with them? $\endgroup$
    – Jeremy Miles
    Dec 29, 2016 at 22:54
  • 1
    $\begingroup$ Internal consistency makes sense only if the items are all caused by the same underlying construct - e.g. if you're depressed, you answer depression items positively, and we score depression by adding up the scores. If you're not creating a score, there's no underlying construct, so no reliability. $\endgroup$
    – Jeremy Miles
    Dec 29, 2016 at 23:24

1 Answer 1

Reset to default
4
$\begingroup$

Currently, Cronbach's alpha is not applicable, either for the instrument as a whole (grouping together all 34 items) or to any subset of items. Alpha requires that each item be scored numerically -- perhaps on a scale such as 1-5, but at least on a "scale" from 0 to 1. With nominal answer choices this is not workable except, possibly, with some additional steps.

Is there some thread of responses running through the various items that, if selected, would express some common opinion or theme? For example, if a preference for Beethoven is reflected in Question 1 Choice A (for short, "Q1-A"), Q2-C, Q3-B, and Q4-B, then you could, as @Jeremy Miles seems to be suggesting, score those responses as "1," and all other responses in those four items as "0." Then the scores of those four items would allow computation of Alpha.

Perhaps some other theme could be given similar treatment, using some other set of items. Then you would have a second scale for which to report alpha.

Without such scoring, it hardly has meaning to seek to measure reliability via internal consistency. There are other types of reliability. In addition to Jeremy's point about test-retest, it is conceivable, if a longshot, that inter-rater reliability would apply, or even parallel-forms.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.