You can handle the issue of missing annotations using a generalized agreement coefficient (see Gwet, 2014). This will basically use all the data you do have. You can handle the issue of multi-label annotations as you suggested, by treating the classes individually. For the imbalanced classes, you have several options. One would be to use a chance-adjusted agreement measure that is sensitive to class distributions (e.g., Cohen's kappa or Scott's pi), although in all likelihood you would just end up with a low score due to the high expected chance agreement (Feinstein & Cicchetti, 1990). Another would be to use category-specific agreement (i.e., positive agreement and negative agreement for binary classes) as described by Cicchetti & Feinstein (1990). Then you look to see how good your agreement is on the positive and negative classes separately; you don't get a single number like accuracy or kappa but you get around the distributional difficulties.
Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.
Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10/fwqv5m
Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. https://doi.org/10/czkxkb