Inferential interpretation of Kappa reliability I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$).
Question: Is the following interpretation technically correct?
After accounting for the chance-expected agreement, we expect ratings from any 90 raters would genuinely agree with one another on 37.5% of these 100 items.
Note: I'm NOT generalizing to any wider populations of raters or items in my interpretation above. I'm only generalizing to OTHER samples of raters OF THE SAME SIZE.
 A: In my opinion, it would be unwise to try to generalize from a sample of 2 raters to a larger population of raters. I would either use this as an estimate of how much agreement there is between these two specific raters or, if you really want to generalize, then collect a larger sample of raters. Along similar lines, it would probably be good to collect more than 10 items as well if this is to be published or is important in some other way. Smaller samples are notoriously unrepresentative. Imagine trying to draw conclusions about Americans in general based on data about just two (or even 10) of them. I would also recommend including some quantification of your uncertainty in the estimate, such as a 95% confidence interval. Finally, you may want to read up on generalizability theory, which provides statistical tools for determining how well an estimate may generalize to different facets (e.g., raters or items) and how many raters or items you would need to achieve a desired level of certainty. 
