Calculate sample size for a moderation exercise, using Cohen's Kappa I need to conduct a moderation study using 3-4 raters, a scale of 5-10 items, and between 1 and 5 samples. Would my Kappa coefficient be meaningful with this sample size? Would I be able to generalise to the population that the scale provides sufficient information to the raters to produce reliable results, consistent value judgements?
 A: The precision of your point estimate (i.e., kappa coefficient) should be estimated using a confidence interval. This will provide a range of plausible values for the population parameter of interest (i.e., the "true" reliability of your raters). If this interval is narrow, then you can argue that the raters' reliability would generalize (assuming your sample was representative of the population).
The width of parametric confidence intervals (i.e., their margins of error) is determined by the sample size and by the amount of variability evident between your samples. You have a small sample size, so that will make the margin of error larger. The variability between samples will be important if you don't have the flexibility to increase the sample size. If all of the samples have very similar point estimates, then you may end up with a narrow confidence interval even with a small sample size. However, if there is considerable variability between samples, then you will end up with a wide confidence interval.
$$CI = M_{\kappa}\pm MOE=M_\kappa\pm SE_{M{_\kappa}}*t_{crit}=M_{\kappa}\pm \frac{s}{\sqrt{n}}*t_{crit}$$
where $M_{\kappa}$ is the mean kappa across all samples, $s$ is the standard deviation of the kappas across samples, $n$ is the number of samples, and $t_{crit}$ is the critical $t$ value with $n-1$ degrees of freedom.
