Inter-rater agreement after rater 1-based sampling Rater 1 rated (+/-) some cases. Rater 2 gets to rate all the Rater 1 + cases, but only half of the rater 1 - cases. Is there a way to account for this sampling scheme when calculating inter-rater agreement (e.g. Cohen's kappa)?
 A: I'm going to assume that, by "account for", you mean to preserve the properties of Cohen's kappa so that it can be interpreted the same way despite this different sampling scheme. I'm also assuming that the half of the Rater 1 $(-)$ cases that Rater 2 does not get to rate are selected at random
Cohen's kappa is defined as
$$\kappa = 1 - \frac{1-p_o}{1-p_e}$$
where $p_o$ is the observed propability of agreement among raters and $p_e$ is the hypothetical probability of rater agreement by random chance.
In the case of $p_o$, you just exclude the cases that were not rated by both raters. $p_e$ is estimated as
$$p_e = p_{1+}p_{2+} + p_{1-}p_{2-} = p_{1+}p_{2+} + (1-p_{1+})(1-p_{2+})$$
where $p_{i+}$ is the estimated probability that Rater $i$ will classify a case as $(+)$. For $p_e$, we need to adjust for the fact that the raters rated different numbers of samples. Let $n_{i+}$ be the number of cases rated $(+)$ by Rater $i$. Then we have $p_{1+} = \frac{n_{1+}}{N}$ and $p_{2+} = \frac{n_{2+}}{\frac{1}{2}N}$.
