0
$\begingroup$

I have a time-series dataset which is annotated by 4 individual annotators where I have following things in annotations.

  1. Its possible that one annotator has not annotated all samples (i.e. missing annotations).
  2. It has multiple classes and one sample can have one or more classes (i.e. multi-label).

I can handle point2 by analyzing each class individually.

My main concern is my data is highly imbalanced i.e. label 0 and label 1 in data might be 95% and 5% (this is just an example and ratio is unknown but definitely very high for 0s) hence agreement probability may be high because of agreement in 0s. What measure might be more reliable here which takes disagreement in account?

$\endgroup$
0
$\begingroup$

You can handle the issue of missing annotations using a generalized agreement coefficient (see Gwet, 2014). This will basically use all the data you do have. You can handle the issue of multi-label annotations as you suggested, by treating the classes individually. For the imbalanced classes, you have several options. One would be to use a chance-adjusted agreement measure that is sensitive to class distributions (e.g., Cohen's kappa or Scott's pi), although in all likelihood you would just end up with a low score due to the high expected chance agreement (Feinstein & Cicchetti, 1990). Another would be to use category-specific agreement (i.e., positive agreement and negative agreement for binary classes) as described by Cicchetti & Feinstein (1990). Then you look to see how good your agreement is on the positive and negative classes separately; you don't get a single number like accuracy or kappa but you get around the distributional difficulties.

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.

Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10/fwqv5m

Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. https://doi.org/10/czkxkb

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.