3
$\begingroup$

Let $M$ be a n x k matrix which is the outcome of a subjective test, where $n$ is the number of samples and $k$ is the number of raters. Values in $M$ can range from 0 to 1 with a step of 0.1. Since the number of samples is high and the evaluation procedure is long, each rater evaluated only a subset of samples. Samples are provided randomly. Therefore, $M$ looks like this:

$$ M = \begin{bmatrix} 0.1 & NaN & 0.2 & \cdots \\ NaN & NaN & 0.15 & \cdots \\ 0.8 & 0.75 & NaN & \cdots \\ \vdots & \vdots & \vdots & \ddots \\ \end{bmatrix} $$

with $NaN$ representing missing data.

What are the available agreement measures for such a test, where some data are missing? The average number of rating for each sample is 5. $n$ is 50 and $k$ is 24. I am interested both in general agreement between raters and agreement on a specific sample. Also, is it possible to find out which of the raters was the less reliable?

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you provide more information about the outcome? From the example it looks like it's a value in range $[0,1]$ $\endgroup$
    – matus
    Mar 27, 2017 at 13:34
  • $\begingroup$ Yes, I added this information. Is there anything else I can include in the description of the problem? $\endgroup$
    – firion
    Mar 28, 2017 at 12:32
  • $\begingroup$ Is the scale fully continuous between 0 and 1, or are only certain points in that range possible (e.g.,0,00, 0.05, ... 1.00)? $\endgroup$
    – Jeffrey Girard
    Apr 9, 2017 at 0:26
  • $\begingroup$ It is a discrete scale with a step equal to 0.1. I am adding this information in the question $\endgroup$
    – firion
    Apr 10, 2017 at 7:59

2 Answers 2

Reset to default
3
$\begingroup$

You will need to use a generalized formula for calculating agreement ($A$) which allows for multiple raters, multiple categories, non-nominal categories, and missing data. The non-nominal categories will be handled using a weighting scheme ($w$) and the missing data will be handled using rater counts ($r$) and by only examining items that had ratings from at least two raters ($n'$). You can then either directly report this agreement measure or adjust it for chance-agreement using one or more estimates of chance agreement (e.g., Cohen's kappa, Scott's pi, Gwet's gamma). I am including here the formula for calculating ratio weights since I assume that is what your scale is. Other weighting schemes (e.g., interval) are also possible. More information, including MATLAB functions, are available on my mReliability website. Kilem Gwet has an excellent textbook on this topic and has made functions available for Excel, SAS, and R on his AgreeStat website.

$$w_{kl}=1-\frac{((x_k-x_l)/(x_k+x_l))^2}{((x_{max}-x_{min})/(x_{max}+x_{min}))^2}$$ $$r_{ik}^\star=\sum_{l=1}^qw_{kl}r_{il}$$ $$A=\frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^q\frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$

where $w_{kl}$ is the weight assigned to one rater assigning an item to category $k$ and the other rater assigning the item to category $l$, $x_k$ and $x_l$ are the numerical values of categories $k$ and $l$, $x_{max}$ and $x_{min}$ are the numerical values of the maximum and minimum categories, $r_{il}$ is the number of raters that assigned item $i$ to category $l$, $q$ is the total number of possible categories, $r_i$ is the number of raters that assigned item $i$ to any category, and $n'$ is the number of items that were rated by two or more raters.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Can you help me relating your formula with my specific case? For example, what are categories? Should I treat scores like categories (e.g. 0 to 0.1 is the first category, 0.1 to 0.2 is the second and so on)? $\endgroup$
    – firion
    Apr 12, 2017 at 9:18
  • $\begingroup$ You had said that you have a scale from 0 to 1 with discrete steps every 0.1. If this is true, then your categories would be 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0. Discrete would mean that only these 11 options are possible. $\endgroup$
    – Jeffrey Girard
    Apr 12, 2017 at 19:36
  • $\begingroup$ Yes you are right. Can you tell me how to interpret the resulting value of A? For example, what does an agreement of 0.5 mean? $\endgroup$
    – firion
    Apr 13, 2017 at 8:57
  • $\begingroup$ With nominal categories, you are looking at each item and calculating the following quotient: the number of pairwise combinations of raters that assigned the item to the same category divided by the number of pairwise combinations of raters that could have assigned the item to the same category. A is the average of this quotient across all items. So A=0.5 would mean that your raters had achieved 50% of the pairwise agreements that were possible. With non-nominal categories, it's the same idea but you now give "partial credit" for raters assigning similar but not identical categories. $\endgroup$
    – Jeffrey Girard
    Apr 13, 2017 at 13:48
  • 1
    $\begingroup$ You can cite: Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics. $\endgroup$
    – Jeffrey Girard
    Oct 11, 2017 at 13:16
1
$\begingroup$

Depending on the scale niveau (here either ordinal or cardinal) of each rater's judgments, you can use either Kendall's coefficent of concordance (W) or the the intraclass correlation procedure. Generalizations of these statistical procedures for the case of randomly missing data are provided by the R-package irrNA. Both procedures allow you to evaluate the reliability of the whole group of raters, as well as the reliabilities of same group, but missing single raters. By comparison of these coefficients, you can guess the contribution of each rater to the group's reliability. (In case of ordinally scaled ratings, use the mean Spearman's rho, not W for this comparison.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.