Calculating inter-annotator agreement Are there situations when it is allowed to omit calculating expected agreement but use only observed agreement as reliable measure? I have multi-label classification (in particular annotation of semantic relations between words in a sentence, so the probability of chance agreement is very low) with multiple annotators.   
 A: Yes; it is perfectly in order to calculate the proportion of agreement and give a confidence interval for it. In fact this is often helpful even in situations where the raters are only using two categories but one of them is very rare. In such cases Cohen's $\kappa$ can be low while proportion of agreement is high, so presenting both gives a better picture. I would suggest giving both in your case too to let the reader see what exactly is happening.
A: There are also different ways to estimate chance agreement (i.e., different models of chance with different assumptions). If you assume that all categories have a relatively equal chance of being guessed at random, then your chance agreement is probably low. But if not, then chance agreement might be estimated as high. For instance, let’s say you have 12 possible categories but in practice only 2 are used with any frequency. Then certain models of chance with provide a higher chance agreement estimate despite there being many categories. My advice would be to estimate and report both observed agreement and expected agreement using different assumptions. This gives more information and insight into performance.
