Reliability of yes or no questionnaire

Question

I have a questionnaire of 15 questions with "yes" or "no" responses distributed to 20 participants. It will be used as an index measure of a variable. How do I measure the reliability of my index?

Answer 1

Some Preliminary Checks

I'm guessing you are looking for a reliability measure for the internal consistency of your items (whether they measure the same thing). You need to consider the following:

1. Are the items measuring the same construct? (For example, you may have 10 questions about anxiety, but 5 relate to performance anxiety and 5 are related to academic anxiety).
2. Do you think each question has two equivalence? In other words, do they all relate to the same latent factor you are measuring equally?
3. Is a dichotomous measure the best way to measure this outcome? Would a Likert scale be better? Or some other continuous outcome?
4. Are some of your items potentially redundant?
5. Do some of your items ask questions in the right way? (for example, are there double negatives that make questions confusing?).

Cronbach's Alpha

After knowing the answer to questions like these, you may consider using a few types of internal reliability checks. One is Cronbach's alpha, which is defined as so:

$$ \alpha = (\frac{k}{k-1})(1-\frac{\Sigma{s^2_i}}{s^2_t}) $$

where $s^2_i$ is the variance of each item, $s^2_t$ is the total variance of all the items, and $k$ is the number of items used. You will notice already this reliability coefficient is dependent on the number of items, which means that this measure can be be artificially inflated by adding several items. In your case, I don't imagine that is an issue, but this measure also requires an assumption of both uni-dimensionality (measures the same construct) and tau equivalence (equally load onto the same factor), which is why I asked earlier about this. For calculating this measure, this video may be useful if you are calculating by hand.

Since your items are binary, this would also technically be calculated by the Kuder-Richardson Formula 20. However, Cronbach's does pretty much the same thing.

McDonald's Omega

Arguably a more thorough but more difficult reliability coefficient you could consider is McDonald's omega. The unidimensional and categorical case, $\omega_{u-cat}$, allows one to estimate binary items. Unlike alpha, it does not assume equal factor loadings but requires a confirmatory factor analysis (CFA) to estimate, which may be difficult if you have not used any form of factor analysis before. If you assume the measure is multi-dimensional, then you may consider hierarchical omega, $\omega_h$, but this has additional complication.

Citations

There are some excellent articles for learning about reliability and how to calculate it. Below are a few good ones.

• Dunn et al., 2013
• Flora, 2020
• Tavakol & Dennick, 2011