If you have multiple raters provide continuous ratings for all items, then it would indeed be sensible to calculate and use the mean of all raters for each item in substantive analyses. There are even methods for estimating the reliability of this mean series (e.g., average score intraclass correlations; McGraw & Wong, 1996). One warning I would offer, however, is that count data (which you seem to describe) is not always distributed normally. In the case of a highly skewed distribution, alternative methods for reliability will be necessary (e.g., a weighted chance-adjusted agreement index).
If you have multiple raters provide categorical ratings for all items, the mean of these ratings will no longer be meaningful in most cases. However, you could still use a heuristic or voting procedure to aggregate ratings (e.g., use the modal category with some a priori tie-breaking procedure). Determining the reliability of the resulting aggregate will be tricky to estimate, but you could mirror the logic used for continuous ratings and compare the aggregate to all individual ratings and average the results.
If, however, you do not have multiple ratings for all items, then it would not make sense to average the results in just the subset for which you do. In this case, you should just pick one rater to use for each item (either randomly or using some explicit rationale).
In my work on nonverbal behavior, I have used all three of these approaches for different projects. When using expert raters on a very time-consuming task (Girard et al., 2014), we opted to use categorical ratings from individual raters for each item and use the most senior rater for those few items that were rated by multiple raters. When using non-expert raters on that same time-consuming task (McDuff, Girard, & el Kaliouby, in press), we opted to use a voting procedure to aggregate the categorical ratings of multiple raters for each item. And finally, when using raters on a quicker and highly inferential task (Ross, Girard, et al., 2016), we opted to average the continuous ratings of six different raters for each item.
References
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014). Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses. Image and Vision Computing, 32(10), 641–647.
McDuff, D., Girard, J. M., & El Kaliouby, R. (in press). Large-scale observational evidence of cross-cultural differences in facial behavior. Journal of Nonverbal Behavior.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
Ross, J. M., Girard, J. M., Wright, A. G. C., Beeney, J. E., Scott, L. N., Hallquist, M. N., … Pilkonis, P. A. (2016). Momentary patterns of covariation between specific affects and interpersonal behavior: Linking relationship science and personality assessment. Psychological Assessment.
