# How to calculate sample size for measuring level of agreement between devices?

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and other correlation statistics (Spearmen or Pearson's correlation). The two devices will be taking measurements for a group of patients on two classifications (grade 0 - grade 4, for example).

So my question is, how would I calculate the sample size? Should it be a regular one or a sample size for Cohen's kappa only? And what would be the exact formula that I should use? Since I have come across lots of formulas but I don't know which to use.

• Without meaning for this to sound aggressive, you need to become clearer on what you want to do. Cohen's kappa is not a good measure of agreement, but can be used to test if there is agreement when the most reasonable default is that there is $0$ agreement & you need to prove otherwise. That doesn't sound like your situation. People use kappa unthinkingly b/c everyone assumes you need p-values. Nov 14, 2023 at 15:05
• If you want to measure the amount of agreement, then determine what aspect of agreement you care about, measure it & get a CI. If you need that CI to be no wider than some amount, then solve for the CI width. If you need to establish that agreement is > some threshold, determine the minimum threshold for the new device to be worthwhile, specify the agreement you believe is true & power for that. Etc. Most likely, these would be done by simulation. Nov 14, 2023 at 15:06
• Could you consider concordance aka Kendall's tau? Or even simple paired-anova? Nov 14, 2023 at 15:31

Correlation is not a very good measure for this sort of thing. Why not? Because you can have high correlation without much agreement. If the new device consistently scores 1 unit higher than the gold standard, the correlation will be perfect but that is not good agreement.

Nor is Cohen's Kappa ideal, because here you have a gold standard.

I think the best method is Bland and Altman's "Limits of Agreement" method, although this is primarily for continuous measures, you might make an argument that your ordinal scale is a representation of an underlying continuous scale. See Wikipedia and references therein; Googling reveals a large literature on this.

Bland and Altman's original paper from 1986 in Lancet, is publicly available as a PDF (it is reference 3 on the Wikipedia list); that article is non-technical but it refers to an earlier paper by the same authors in The Statistician. Sample size is not mentioned there; deliberately, I believe.

I would say that you want enough observations so that the plots that B and A recommend are reasonable. Of course, this involves judgement rather than a formula, but ... Data analysis generally does involve judgement.

An alternative approach would be to power your study as an equivalence study or a non-inferiority study. In this situation you would see if you can establish that the level of disagreement between the two methods is within a certain (clinically meaningful) margin.

It depends what your goal is. If you goal is to potentially replace an existing device with a new one then establishing equivalence (or non-inferiority) may be sufficient.

In the case of paired binary outcomes see: Liu, Jen‐pei, et al. "Tests for equivalence or non‐inferiority for paired binary data." Statistics in medicine 21.2 (2002): 231-245.

The following R code implements equation (7) from the above paper.

diag_test_noninf <- function(p01 = 0.3,
delta = 0.25,
z_alpha = qnorm(0.05),
z_beta2 = qnorm(0.20/2),
w=1){

n <- ceiling(2*p01*(((z_alpha + z_beta2)/delta)^2))

cat("The total number of samples required is", n, "\n")

return(n)

}