I'm analyzing a study where participants needed to list as many plausible solutions to a given question. There are two raters who are counting the number of plausible solutions, what is the best way to conduct an inter-rater reliability analysis? There are two independent raters. Ultimately the study is trying to see if there is a difference in the number of plausible solutions generated between two conditions.

Also, is it necessary that all cases be rated by both and reliability analyzed or can just a percentage of the cases? For example, if reliability was analyzed for just the pilot data (with a sample of just over 100 participants), would it then be okay to use only one rater for the larger study (of close to 400 participants per a power analysis) if there was high inter-rater agreement?


1 Answer 1


You should use a chance-corrected index of categorical agreement with ratio weights; I would recommend the generalized form of Bennett et al.'s S score. The formula would be:

$$S=\frac{p_o-p_c}{1-p_c}$$ $$p_o=\frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^q\frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$ $$p_c=\frac{1}{q^2}\sum_{k,l}w_{kl}$$ $$r_{ik}^\star=\sum_{l=1}^qw_{kl}r_{il}$$ $$w_{kl}=1-\frac{[(x_k-x_l)/(x_k+x_l)]^2}{[(x_\max-x_\min)/(x_\max+x_\min)]^2}$$

where $q$ is the total number of categories,

$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$,

$r_{il}$ is the number of raters that assigned item $i$ to category $l$,

$n'$ is the number of items that were coded by two or more raters,

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$,

$r_i$ is the number of raters that assigned item $i$ to any category, and

$x_k$ is the actual numerical value of category $k$.

You can calculate this value automatically if you format the data correctly and use my mSSCORE function in MATLAB or Kilem Gwet's agree.coeff3.raw.r function in R (the bp.coeff.raw subfunction).

It is not necessary that all cases be rated by both raters, but it would be preferable to have as many as possible rated by both. Including more participants will yield a narrower confidence interval on the reliability estimate, making it more informative and trustworthy. You also need to be able to argue convincingly that the participants rated by a single rater are no different than those rated by multiple raters (and that therefore the known reliability in the latter is representative of the unknown reliability in the former). The best way to do this would be to randomly assign participants to be rated by one or both raters and to do this throughout the project rather than doing an initial reliability test and then assuming that that is the reliability that will continue forever. For more introductory information about observational measurement, please see my recent article on the topic.

  • $\begingroup$ First, thanks so much for your response! Unfortunately, I'm not sure that this index will work for my data since the raters are not assigning categories. Instead, participants are randomly assigned to one of two conditions as the IV, and for the DV raters are simply counting the number of proper solutions provided by each respondent. However, I just want to make sure to address possible subjectivity in terms of what raters are agreeing is a proper solution. $\endgroup$
    – CCC
    Jul 3, 2016 at 3:29
  • 1
    $\begingroup$ Sorry if it was unclear, but "counts" can be considered categories. For instance, assigning a count of 1 is like assigning it to a category called "one." The only difference is that the counts have a ratio relationship as opposed to a nominal one. But this relationship is what the ratio weights are for. So unless I'm really misunderstanding, this solution will work. $\endgroup$ Jul 3, 2016 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.