3
$\begingroup$

The sample size computation

require(irr)
N.cohen.kappa(rate1 = 0.9, rate2 = 0.5, k1 = 0.7, k0 = 0.5, alpha=0.05, power=0.8, twosided=FALSE)

successfully runs and gives a result ... but shouldn't it fail with some kind of warning?

Consider the pattern of missing values in Table 1 and the second sentence of the left-bottom paragraph of the second page here, which reflect an upper bound for Kappa when rate1 is not the same as rate2. For example, Kappa must be less than 0.2 if rate1 = 0.9 and rate2 = 0.5, right? So shouldn't the function fail when I use those two rates with k1 $ = 0.7$, since that "true kappa" is not feasible?

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, it should fail with a warning. The reasons cited in the OP are correct; here I'll simply derive an upper bound for kappa that proves the point.

Define several "true" population level parameters:

  • $r_1=$ rate1 and $r_2=$ rate2 are the rates at which the two raters select the target category
  • $\pi_e$ is the expected rate of agreement between the two raters that you get if you assume the operate independently. We can write this explicitly in terms of $r_1$ and $r_2$. The probability they both select the target category is $r_1 r_2$ and the probability they both don't select the target category is $(1-r_1)(1-r_2)$. Thus, in total, $\pi_e = r_1 r_2 + (1-r_1)(1-r_2) = 1 - r_1 - r_2 + 2r_1 r_2$.
  • $\pi_0$ is the rate of agreement between the two raters, i.e., the fraction of cases in which they agree with each other.

Now we have

$$\kappa = \frac{\pi_0 - \pi_e}{1- \pi_e} = \frac{\pi_0 - 1 + r_1 + r_2 - 2r_1 r_2}{r_1 + r_2 - 2r_1 r_2}$$

Observe that the maximum rate of agreement between the two raters on selecting the target category is $\min(r_1, r_2)$. Similarly, the maximum rate of agreement on not selecting the target category is $\min((1-r_1), (1-r_2))$. In total, $\pi_0 \leq \min(r_1, r_2) + \min((1-r_1), (1-r_2))$.

Assume, WLOG, that $r_1 \geq r_2$. Then $\pi_0 \leq r_2 + 1-r_1$. Thus

$$ \kappa \leq \frac{(r_2 + 1-r_1) - 1 + r_1 + r_2 - 2r_1 r_2}{r_1 + r_2 - 2r_1 r_2} = \frac{2r_2 - 2r_1 r_2}{r_1 + r_2 - 2r_1 r_2}= 2r_2 \frac{1 - r_1}{r_1 + r_2 - 2r_1 r_2} $$

So, what happens when we plug in $r_1 = 0.9$ and $r_2 = 0.5$ as in the OP? Then $\kappa$ can't be greater than $0.2$. Indeed, the power computation in the OP rests on the false premise that the true kappa is 0.7, and so it would be desirable for the function to throw a warning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.