I recently got some peer-review edits back on a manuscript validating a binary molecular detection assay protocol (accepted, subject to peer reviewers approval). One of the results was from an inter-laboratory portability assessment where certified negative clinical specimens were spiked with known amounts of the target pathogen near the lower detection limit and sent to a collaborating lab for blinded testing with the new protocol (a third of the samples tested were un-spiked, negative control specimens). In the manuscript, I reported simple agreement percentages (positive, negative, and overall) between the testing results and the expected results based on whether they were spiked or not. These were calculated using this online calculator (the Klopper-Pearson intervals were also reported).
However, in the suggested edits, a statistical reviewer suggested reporting Cohen's Kappa instead of simple agreement. Based on my literature search (including regulatory applications), simple agreement percentage is the standard reporting metric for similar detection assays. I have no objection to deviating from that if it's valid, but I'm not so sure it is. First, the purpose wasn't to describe testing agreement between two labs, but to show that another lab, with different personnel and equipment, could follow the protocol and still accurately identify positive and negative specimens (i.e. assay portability). Second, the samples are contrived specimens, not true clinical specimens. As such, the data in question really only comes from single "rater," since only one lab tested the samples in question. Lastly, they aren't necessarily representative of true specimens in terms of pathogen load or prevalence. (For various reasons, we are not able to get true clinical specimens, so we're just trying to publish the protocol so that someone with better clinic access can take it up where we left off).
Since the reviewer didn't provide a justification, I can only speculate why they expect to see one statistic vs the other. I guess I'm just asking if I'm overlooking something important in deciding to stick with simple agreement percentages, and if my above explanations are likely to satisfy a peer-reviewer in a reply letter. I could also point out that all of the relevant numbers are in the manuscript, such that any reader would be able to calculate their preferred agreement statistic.