Should Cohen's Kappa or simple percentage agreement be used in molecular assay validation with contrived samples?

Question

I recently got some peer-review edits back on a manuscript validating a binary molecular detection assay protocol (accepted, subject to peer reviewers approval). One of the results was from an inter-laboratory portability assessment where certified negative clinical specimens were spiked with known amounts of the target pathogen near the lower detection limit and sent to a collaborating lab for blinded testing with the new protocol (a third of the samples tested were un-spiked, negative control specimens). In the manuscript, I reported simple agreement percentages (positive, negative, and overall) between the testing results and the expected results based on whether they were spiked or not. These were calculated using this online calculator (the Klopper-Pearson intervals were also reported).

However, in the suggested edits, a statistical reviewer suggested reporting Cohen's Kappa instead of simple agreement. Based on my literature search (including regulatory applications), simple agreement percentage is the standard reporting metric for similar detection assays. I have no objection to deviating from that if it's valid, but I'm not so sure it is. First, the purpose wasn't to describe testing agreement between two labs, but to show that another lab, with different personnel and equipment, could follow the protocol and still accurately identify positive and negative specimens (i.e. assay portability). Second, the samples are contrived specimens, not true clinical specimens. As such, the data in question really only comes from single "rater," since only one lab tested the samples in question. Lastly, they aren't necessarily representative of true specimens in terms of pathogen load or prevalence. (For various reasons, we are not able to get true clinical specimens, so we're just trying to publish the protocol so that someone with better clinic access can take it up where we left off).

Since the reviewer didn't provide a justification, I can only speculate why they expect to see one statistic vs the other. I guess I'm just asking if I'm overlooking something important in deciding to stick with simple agreement percentages, and if my above explanations are likely to satisfy a peer-reviewer in a reply letter. I could also point out that all of the relevant numbers are in the manuscript, such that any reader would be able to calculate their preferred agreement statistic.

Answer 1

Percentage agreement or accuracy tends to not be a universally good choice of metric because it is very sensitive to the underlying class distribution. For example, imagine that the secondary lab doesn't actually do any testing at all, but just sends back a classification of "Positive" for every sample. If you send them a balanced mix of positives and negatives, they'll have 50% accuracy or 50% agreement with your labels. If you send them just a box of positive samples, however, they will be 100% in agreement with your labels. If you send them a box of negative samples, they will be 0% accurate. Even though the lab is doing absolutely nothing useful in all of these circumstances, you can produce wildly different figures of percentage agreement based entirely on the content of what you send them.

Cohen's kappa does not have this problem because it effectively corrects for chance agreement between raters. If both labs say the positive/negative split is 50/50, you'd expect 50% of the samples to be labeled the same by chance alone. If the split is 90/10, however, you'd expect 82% of the labels to be identical even with random guessing. Without examining the underlying class split, it's usually impossible to know whether a percentage agreement figure is better than random guessing or not.