Agreement among raters with missing data Let $M$ be a n x k matrix which is the outcome of a subjective test, where $n$ is the number of samples and $k$ is the number of raters. Values in $M$ can range from 0 to 1 with a step of 0.1. Since the number of samples is high and the evaluation procedure is long, each rater evaluated only a subset of samples. Samples are provided randomly. Therefore, $M$ looks like this:
$$
M = \begin{bmatrix} 0.1 & NaN & 0.2 & \cdots \\ 
                    NaN & NaN & 0.15 & \cdots \\
                    0.8 & 0.75 & NaN & \cdots \\
                    \vdots & \vdots & \vdots & \ddots \\  
\end{bmatrix}
$$
with $NaN$ representing missing data.
What are the available agreement measures for such a test, where some data are missing? The average number of rating for each sample is 5. $n$ is 50 and $k$ is 24. I am interested both in general agreement between raters and agreement on a specific sample. Also, is it possible to find out which of the raters was the less reliable?
 A: You will need to use a generalized formula for calculating agreement ($A$) which allows for multiple raters, multiple categories, non-nominal categories, and missing data. The non-nominal categories will be handled using a weighting scheme ($w$) and the missing data will be handled using rater counts ($r$) and by only examining items that had ratings from at least two raters ($n'$). You can then either directly report this agreement measure or adjust it for chance-agreement using one or more estimates of chance agreement (e.g., Cohen's kappa, Scott's pi, Gwet's gamma). I am including here the formula for calculating ratio weights since I assume that is what your scale is. Other weighting schemes (e.g., interval) are also possible. More information, including MATLAB functions, are available on my mReliability website. Kilem Gwet has an excellent textbook on this topic and has made functions available for Excel, SAS, and R on his AgreeStat website.
$$w_{kl}=1-\frac{((x_k-x_l)/(x_k+x_l))^2}{((x_{max}-x_{min})/(x_{max}+x_{min}))^2}$$
$$r_{ik}^\star=\sum_{l=1}^qw_{kl}r_{il}$$
$$A=\frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^q\frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$
where $w_{kl}$ is the weight assigned to one rater assigning an item to category $k$ and the other rater assigning the item to category $l$, $x_k$ and $x_l$ are the numerical values of categories $k$ and $l$, $x_{max}$ and $x_{min}$ are the numerical values of the maximum and minimum categories, $r_{il}$ is the number of raters that assigned item $i$ to category $l$, $q$ is the total number of possible categories, $r_i$ is the number of raters that assigned item $i$ to any category, and $n'$ is the number of items that were rated by two or more raters.
A: Depending on the scale niveau (here either ordinal or cardinal) of each rater's judgments, you can use either Kendall's coefficent of concordance (W) or the the intraclass correlation procedure. Generalizations of these statistical procedures for the case of randomly missing data are provided by the R-package irrNA.
Both procedures allow you to evaluate the reliability of the whole group of raters, as well as the reliabilities of same group, but missing single raters. By comparison of these coefficients, you can guess the contribution of each rater to the group's reliability. (In case of ordinally scaled ratings, use the mean Spearman's rho, not W for this comparison.)
