I am very new to statistics, and I need help determining the the best inter-reliability test for a questionnaire. My questionnaire will be assessed by three raters. I'm just providing the sample of my questionnaire and what it can look like:

dat = data.frame(rater1 = 
                 rater2 = 
                 rater3 = 

What is the best way to assess inter-reliability for mixed data types questionnaire? And how can I tell where the most disagreement comes from? for example is it #1 where rater1=3 rater2=4, rater3=1. How Can I tell this is the question that is leading to the most difference?

  • $\begingroup$ One simple way to handle the mixed data-types is to just dummy-code Y = 1 and N = 0, and then do intra-class correlation between the raters. I'm thinking since most of the data is continuous. $\endgroup$ Commented Dec 12, 2023 at 23:53
  • $\begingroup$ I'm not sure if that is valid for all methods. I just have limited knowledge about this topic, but coefficient's like Cohen's kappa and Gwet's AC1 do assume equal categories. These coefficients are normally also calculated for every question, so with that you can see which questions are (not) problematic. $\endgroup$ Commented Dec 13, 2023 at 9:11
  • $\begingroup$ It may be helpful to know why some items are binary (Y/N) and some are slightly more discrete (I'm guessing Likert scale). Seems this pointlessly restricts the variance of the last items and complicates matters, but perhaps there is a reason. $\endgroup$ Commented Dec 15, 2023 at 13:58
  • $\begingroup$ The way the questionnaire is designed is to have Y/N question at the end of each section, and whether the assessor thinks the criteria has been satisfied. $\endgroup$
    – Ashti
    Commented Dec 15, 2023 at 15:23

1 Answer 1


Modelling Annotators' Reliability and Agreement

Note that I am using "annotator" as a synonym to "rater", and "observer". Reliability and agreement are synonyms with certainty and trust. Thus antonyms are uncertainty or variance. Other keywords of interest for searching: crowdsource aggregation, crowdsource truth inference or truth discovery.

To your first question, "the best way" is highly dependent on the specific problem and your data sample is not enough to answer that. As I will note throughout my answer, there are different ways to model things, and in the end you'll have to make an informed decision that best fit your case.

Model Each Question's Space of Possible Answers

First, look at each question in the questionnaire and specify the space of possible answers for each. This is important as it denotes the possible answers for each question and will inform you what kind of probability distribution is proper to use.

If there were a finite set of possible answers, then a categorical distribution is an appropriate distribution to use. If there are other constraints across the possible values that another probability distribution assumes, then that distribution is a better fit in that case. In the case of only two values such as 'Y'/'N', a Bernoulli distribution serves well.

Rarely will you ask people to give an unconstrained real (continuous) number as an answer, but in the case you do, then your space must model the answer as a real number. Typically surveys use a Likert scale or other discretization of the real line to a set of finite bins.

Model Answer Agreement Independent of Annotators' Other Answers

After you have the answer spaces, you can model a distribution over that answer space informed by any prior information, such as a prior distribution or model choice, and the observed answers. You get to chose how to balance their role in informing the resulting distribution. Most people use the empirical distribution/process which has no prior knowledge, and may fit your case. However the empirical process provides asymptotic convergence to truth, which involves infinite samples, not three samples (answers per question) as you have in your example. So, you may also consider Bayesian modeling approaches that involve a prior distribution to incorporate more variance depending on your prior belief of agreement (certainty) or disagreement (uncertainty) amongst answers to that question.

In this case, you can look at each question's answers independent of the annotators' answers to other questions, thus disregarding the reliability of individual annotators. This approach is simpler, and probably more often used, although correctness of this approach is up to the data design(er) and analyzer of this data.

How can I tell where the most disagreement comes from?

Using this approach, the question with the most uncertainty (entropy) or variance is the question that is the source of the most disagreement between all questionnaires.

One way, assuming categorical answers, is to model the answers using the empirical distribution, which results in a categorical. The categorical's entropy may then be analytically computed, as follows $$H(X) = -\sum_{x \in \mathscr{X}} P(X=x) \log P(X=x)$$ where the $\mathscr{X}$ is the space of possible answers and the probability of the modeled categorical random variable $X$ is simply informed by the probability vector found from the empirical distribution of frequencies.

Note that to get a better estimate of the population of annotator's responses you may need more annotators. In your example, the possible probability vectors is severely limited by the total of 3 answers. So $[\frac{1}{3}, \frac{1}{3}, \frac{1}{3}, ...]$ is the most uncertain probability vector possible where the rest of the possible answers, if greater than 3, have probability zero. Due to this, it is often a good idea to have enough annotators to enable a fine grained enough probability vector to approximate the true distribution of answers.

Model Annotator Reliability per Annotator

As mentioned earlier, sometimes we are able to model the individual annotator's agreement over all their answers to other annotators, or with respect to ground truth questions for which we assume to know the correct answer to. This is useful when you have possibly unreliable annotators taking your survey or annotating your data, whether due to maliciousness or due differences in expertise on the annotation task.

This is only able to be done if the unique annotators are labeled and their annotations are associated with their label, which can still be anonymized. In your case, each instance of a questionnaire serves as an annotator identifier. However, if you allow people to retake the questionnaire, then you must provide them with a consistent indicator label across their separate questionnaires, otherwise you will incorrectly treat their responses as coming from different people.

By modelling each annotator's reliability overall, you can then incorporate the certainty in their annotations into each question they answer and thus have a better model of the probability distribution of pooled answers.

One approach is the 1979 David & Skene EM Algorithm [1]. EM stands for Expectation Maximization.

How can I tell where the most disagreement comes from?

Using this approach, you can still find the most disagreed upon question, although it may be different than when modeling only agreement of answers per question in isolation.

In this case though, you are able to find the annotator(s) who disagrees the most with all other annotators. You can also better characterize the agreement. For example, in a group of three annotators, say they all mostly agree for some questions, but you find two questions to have one annotator strongly disagree. With this approach, you can find those questions as well as figure out if the disagreement is due to the same annotator disagreeing from the majority each time, or due different annotators for the different questions.

The Model Complexity Can Grow with Multiple Tasks

When the annotation task consists of multiple tasks that a single annotator may have different expertise within each, it can be important to model the annotator's reliability in each task rather than overall, as we discussed above. This is more complex and nuanced, but at times is desirable and doable if the questions have either tags of which tasks they relate to in a binary yes/no fashion or a weighting of their relation to a task, or some other structural relationship. I leave this to further reading in the following surveys, if desired.

There have been more approaches to modeling individual annotators and their agreement on problems to better understand the information involved in the annotations and to go beyond mere majority or plurality vote. Here are some surveys/reviews:


Closed Access. = Closed Access. Open Access. = Open Access.

  1. Dawid, Alexander Philip and Skene, A. M. "Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm". 1979. Journal of the Royal Statistical Society: Series C (Applied Statistics). Vol 28. pg 20--28. ISSN: 1467-9876. DOI: 10.2307/2346806.
  2. Paun, Silviu and Carpenter, Bob and Chamberlain, J. D. and Hovy, Dirk and Kruschwitz, Udo and Poesio, Massimo. "Comparing Bayesian Models of Annotation". 2018. Transactions of the Association for Computational Linguistics. Vol 6. pg. 571--585.
  3. Zheng, Yudian and Li, Guoliang and Li, Yuanbing and Shan, Caihua and Cheng, Reynold. "Truth Inference in Crowdsourcing: Is the Problem Solved?" 2017. Proc. VLDB Endow. Vol. 10, No. 5. pg. 541--552. ISSN 2150-8097. DOI 10.14778/3055540.3055547.
  4. Miller, Nolan and Resnick, Paul and Zeckhauser, Richard. "Eliciting Informative Feedback: The Peer-Prediction Method". 2005. Management Science. Vol. 51, No. 9. pg. 1359--1373

Further Reading


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.