1
$\begingroup$

I am trying to implement the agreement score illustrated here. In particular, Equation 1 states that the contribution of sample $i$ to the agreement score of rater $j$ is

$$ \hat{s}_{ij} = \frac{n_{i, g_{ij}} - 1}{(\sum_{g=1}^3n_{i, g}) - 1} $$

where $n_{i,g}$ is the observed number of raters giving grade $g$ to sample $i$. In addition, $g_{i,j}$ is the observed grade assigned to sample $i$ by rater $j$, but $n_{i, g_{ij}}$ is not defined. What could be the correct formulation?

The agreement score for rater $j$ is then estimated as

$$ \hat{s}_{j} = \frac{1}{N_j}\sum_{i=1}^m\hat{s}_{ij} $$

where $m$ is the number of samples.

$\endgroup$
0
$\begingroup$

$n_{i,g_{ij}}-1$ is the number of raters who assigned the same grade (namely $g_{ij}$) to the sample $i$ as rater $j$. $n_{i,g_{ij}}$ is then the number of raters who assigned the same grade to the sample $i$ as rater $j$ including rater $j$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.