I'm conducting a study in which I'm testing interrater agreement on a new radiologic technique to the pathological evaluation (gold standard). I've got 58 patients and 15 raters. Raters have been asked to evaluate the degree of cancer invasion ('Deep' vs. 'Superficial') and have also reported how certain they are in their assessment, using a Visual Analog Scale ranging 0-100.

After having calculated specificity, sensitivity and accuracy, and done McNemars test, Cohens kappa, Fleiss kappa and Gwets AC1, I now what to check for association between raters certainty and their accuracy. I've thus created a new variable var_accurate for each rater, coded 0=wrong, 1=accurate. I want to test if this variable is associated with certainty.

This question have been asked before in this forum (How do I study the "correlation" between a continuous variable and a categorical variable? and Correlation between a nominal (IV) and a continuous (DV) variable) so I know the best approach is a point-biserial correlation (https://statistics.laerd.com/spss-tutorials/point-biserial-correlation-using-spss-statistics.php).

Although, my data does not meet the test assumptions. For some examiners (but not all) there is outliers in their certainty levels when split on var_accurate, my data is not normally distributed for all examiners since I get a significant p-value on Shapiro-Wilk test and for some examiners I don't have equal variances when certainty levels are split on var_accurate with significant Lavene's test. Thus, all assumptions are voided for at least a few examiners, not always the same and not always are all assumptions not met for one particular rater.

How do I go on from here?

I'm also wondering if it is correct to do point-biserial correlation for each rater, like analogous to Cohens? Is there a multiple rater test, like analogous to Fleiss kappa for point-biserial correlation?

And why is point-biserial correlation recommended over one-way ANOVA and even Students t-test, for my application, the latter being much more well know?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.