Cohen's kappa often has an asymmetric sampling distribution, particularly in small samples or when the kappa value is close to 0 or 1. This can cause confidence intervals based on standard errors (which assume normality) to be unreliable and prone to undercoverage.
In contrast, for Pearson's correlation coefficient (rho), the Fisher z-transformation is commonly used to stabilize the variance and normalize the sampling distribution, making it easier to calculate accurate confidence intervals. This works well for rho because the z-transformed distribution becomes roughly normal, especially with larger sample sizes.
Unfortunately, there isn’t a simple, widely accepted equivalent of the Fisher z-transformation for Cohen's kappa. Kappa is more complicated because it accounts for agreement due to chance, and its sampling distribution doesn’t easily lend itself to a transformation like rho’s does. While some variance-stabilizing transformations for kappa have been proposed, they are complex and not frequently used in practice.
Given these challenges, bootstrap methods offer a practical and reliable solution. Bootstrapping doesn’t rely on normality assumptions and naturally handles the asymmetry and boundaries of kappa’s sampling distribution. By bootstrapping kappa and constructing confidence intervals from the empirical distribution of the bootstrap estimates, you better reflect the actual shape of the sampling distribution, resulting in more accurate coverage.
The bias-corrected and accelerated (BCa) bootstrap is particularly useful since it adjusts for both bias and skewness in the bootstrap distribution. The BCa method yields asymmetric confidence intervals that respect kappa’s bounds, making it especially helpful for small samples or when kappa is near the extremes of 0 or 1.
