4
$\begingroup$

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and other correlation statistics (Spearmen or Pearson's correlation). The two devices will be taking measurements for a group of patients on two classifications (grade 0 - grade 4, for example).

So my question is, how would I calculate the sample size? Should it be a regular one or a sample size for Cohen's kappa only? And what would be the exact formula that I should use? Since I have come across lots of formulas but I don't know which to use.

Improve this question
$\endgroup$
3
  • $\begingroup$ Without meaning for this to sound aggressive, you need to become clearer on what you want to do. Cohen's kappa is not a good measure of agreement, but can be used to test if there is agreement when the most reasonable default is that there is $0$ agreement & you need to prove otherwise. That doesn't sound like your situation. People use kappa unthinkingly b/c everyone assumes you need p-values. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 14, 2023 at 15:05
  • $\begingroup$ If you want to measure the amount of agreement, then determine what aspect of agreement you care about, measure it & get a CI. If you need that CI to be no wider than some amount, then solve for the CI width. If you need to establish that agreement is > some threshold, determine the minimum threshold for the new device to be worthwhile, specify the agreement you believe is true & power for that. Etc. Most likely, these would be done by simulation. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 14, 2023 at 15:06
  • $\begingroup$ Could you consider concordance aka Kendall's tau? Or even simple paired-anova? $\endgroup$
    – J. Doe.
    Commented Nov 14, 2023 at 15:31

3 Answers 3

Reset to default
11
$\begingroup$

Correlation is not a very good measure for this sort of thing. Why not? Because you can have high correlation without much agreement. If the new device consistently scores 1 unit higher than the gold standard, the correlation will be perfect but that is not good agreement.

Nor is Cohen's Kappa ideal, because here you have a gold standard.

I think the best method is Bland and Altman's "Limits of Agreement" method, although this is primarily for continuous measures, you might make an argument that your ordinal scale is a representation of an underlying continuous scale. See Wikipedia and references therein; Googling reveals a large literature on this.

Bland and Altman's original paper from 1986 in Lancet, is publicly available as a PDF (it is reference 3 on the Wikipedia list); that article is non-technical but it refers to an earlier paper by the same authors in The Statistician. Sample size is not mentioned there; deliberately, I believe.

I would say that you want enough observations so that the plots that B and A recommend are reasonable. Of course, this involves judgement rather than a formula, but ... Data analysis generally does involve judgement.

Improve this answer
$\endgroup$
2
$\begingroup$

An alternative approach would be to power your study as an equivalence study or a non-inferiority study. In this situation you would see if you can establish that the level of disagreement between the two methods is within a certain (clinically meaningful) margin.

It depends what your goal is. If you goal is to potentially replace an existing device with a new one then establishing equivalence (or non-inferiority) may be sufficient.

In the case of paired binary outcomes see: Liu, Jen‐pei, et al. "Tests for equivalence or non‐inferiority for paired binary data." Statistics in medicine 21.2 (2002): 231-245.

The following R code implements equation (7) from the above paper.

diag_test_noninf <- function(p01 = 0.3,
                             delta = 0.25,
                             z_alpha = qnorm(0.05),
                             z_beta2 = qnorm(0.20/2),
                             w=1){
  
  n <- ceiling(2*p01*(((z_alpha + z_beta2)/delta)^2))
  
  cat("The total number of samples required is", n, "\n")
  
  return(n)
  
}
Improve this answer
$\endgroup$
0
$\begingroup$

sorry to be late on this - I am interested in the alternative suggestions of Bland Altman or equivalence of the mean. I would have thought equivalence of the mean was less appropriate as the measuring devices might well measure the same on average but one measures low when the other measures high and vice versa so they don't agree at all. I agree this is an extreme example but even so we can have poor agreement of the paired values but overall equal means. In such a case the SD of the differences would be high but with a big enough sample size equivalence of means can be shown.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.