# Reliability testing for questionnaire with nominal multiple choice questions (one possible answer)

I have used a questionnaire to collect data from 200 people.

This questionnaire was obtained from a published research paper in which researchers had used it in a specific population. It involves 34 items that are nominal multiple choice questions with one possible answer.

For example: Which factor do you consider?

1. X
2. Y
3. Z

Questions do not have all the same number of possible answers. For example, question1 has 3 possible answers, question2 4 and so on.

According to the paper, Cronbach's $\alpha$ was 0.62.

I wanted to use the questionnaire in a different population. I carried out a pilot study in 20 people and made some changes. My colleague in charge of the statistics calculated Cronbach's $\alpha$. It was 0.65. When I finished the study, gathering data from 200 people, the calculated Cronbach's $\alpha$ was 0.57.

However, I myself think that Cronbach's $\alpha$ is not suitable in this case, as my questionnaire is not Likert, nor scaling. I believe that neither was it suitable in the original published paper. Unfortunately, time has passed and test-retest cannot be performed.

So, do you think that Cronbach's $\alpha$ can be applied in this questionnaire? Which is the correct way of coding these data (different coding results in a different Cronbach's $\alpha$)? What other reliability tests can be performed in this case?

• Is there one answer that is correct, and one that is not correct? How is the questionnaire scored? Commented Dec 29, 2016 at 1:58
• Each question has multiple possible answers, but people taking the survey have to choose only one. Questions investigate patients' personal opinion for something. Commented Dec 29, 2016 at 5:15
• And how is it scored? Commented Dec 29, 2016 at 15:09
• If it's not scored, there is no meaning to reliability, except the reliability of individual items (which you could assess with retesting). But don't you then end up with 34 variables? What do you do with them? Commented Dec 29, 2016 at 22:54
• Internal consistency makes sense only if the items are all caused by the same underlying construct - e.g. if you're depressed, you answer depression items positively, and we score depression by adding up the scores. If you're not creating a score, there's no underlying construct, so no reliability. Commented Dec 29, 2016 at 23:24