# Inter-rater agreement after rater 1-based sampling

Rater 1 rated (+/-) some cases. Rater 2 gets to rate all the Rater 1 + cases, but only half of the rater 1 - cases. Is there a way to account for this sampling scheme when calculating inter-rater agreement (e.g. Cohen's kappa)?

I'm going to assume that, by "account for", you mean to preserve the properties of Cohen's kappa so that it can be interpreted the same way despite this different sampling scheme. I'm also assuming that the half of the Rater 1 $$(-)$$ cases that Rater 2 does not get to rate are selected at random

Cohen's kappa is defined as $$\kappa = 1 - \frac{1-p_o}{1-p_e}$$ where $$p_o$$ is the observed propability of agreement among raters and $$p_e$$ is the hypothetical probability of rater agreement by random chance.

In the case of $$p_o$$, you just exclude the cases that were not rated by both raters. $$p_e$$ is estimated as $$p_e = p_{1+}p_{2+} + p_{1-}p_{2-} = p_{1+}p_{2+} + (1-p_{1+})(1-p_{2+})$$ where $$p_{i+}$$ is the estimated probability that Rater $$i$$ will classify a case as $$(+)$$. For $$p_e$$, we need to adjust for the fact that the raters rated different numbers of samples. Let $$n_{i+}$$ be the number of cases rated $$(+)$$ by Rater $$i$$. Then we have $$p_{1+} = \frac{n_{1+}}{N}$$ and $$p_{2+} = \frac{n_{2+}}{\frac{1}{2}N}$$.

• Thank's for the answer. I believe there is an issue with the calculation for $p_0$ but that may because my definition of 'account for' was not clear. Your reaction did guide me to an approach (to long for a comment so made it an answer), would you agree to what I propose there? Jun 15 '20 at 8:27
• Yes, based on your precise definition of "account for", I agree with your approach. Jun 16 '20 at 22:56

@eyeExWhy 's answer directed me to a slightly different approach (unless I misunderstood that answer). I think the error is in 'In the case of $$p_0$$, just excluding the cases that were not rated by both raters'

With 'account for', I mean that knowing the sampling scheme, I want a calculation that in expectation has the same value as when the sampling did not occur.

Start from the un-sampled situation:

$$\begin{bmatrix} & T_{2+} & T_{2-} \\ T_{1+} & 200 & 100 \\ T_{1-} & 100 & 200 \end{bmatrix}$$

Then \begin{align} p_0 & = \frac{200+200}{200+100+100+200} = 2/3 \\ p_{1+} &= p_{1-} = p_{2+} = p_{2-} = 0.5 \\ p_e &= 0.5*0.5+0.5*0.5 = 0.5\\ \kappa & = 1- \frac{1-2/3}{1-0.5} = 1/3 \end{align}

After the sampling, on average we get this (sample the first row with probability 0.5, sample the second row with probability 1)

$$\begin{bmatrix} 100 & 50 \\ 100 & 200 \end{bmatrix}$$

By assigning weights inversely proportional to the sampling probabilities, the sampling scheme is accounted for:

\begin{align} w_{++} &= w_{+-} = 1/0.5=2\\ w_{-+} &= w_{--} = 1/1 = 1 \\ p_0 & = \frac{2*100+200}{2*100+2*50+100+200} = 2/3 \\ p_{1+} &= \frac{2*100 + 2*50}{2*100+2*50+100+200} = 0.5 \\ p_{1-} &= \frac{100 + 200}{2*100+2*50+100+200} = 0.5 \\ p_{2+} &= \frac{2*100 + 100}{2*100+2*50+100+200} = 0.5 \\ p_{2-} &= \frac{2*50 + 200}{2*100+2*50+100+200} = 0.5 \\ p_e &= 0.5*0.5+0.5*0.5 = 0.5\\ \kappa & = 1- \frac{1-2/3}{1-0.5} = 1/3 \end{align}