After calculating inter-rater reliability, which data (e.g. consensus data?) should be used for further analyses? My study used 2 raters in a fully crossed design to code 1) a categorical variable and 2) a continuous variable (i.e counting the number of times something happened). 
My question is, after performing inter-rater reliability and determining the level of agreement between raters to be acceptable, which data should be used for further analyses (T-tests, Anovas, etc)? I have not been able to find a clear answer to this. For the continuous variable, I can see how taking the average would make sense, but is this actually an accepted method? Further, how do you come to a consensus data set for the categorical data?
Or, on the other hand, should the same analyses be performed with both sets of data (data from rater-1 and data from rater-2)?
Thank you for your thoughts!
 A: If you have multiple raters provide continuous ratings for all items, then it would indeed be sensible to calculate and use the mean of all raters for each item in substantive analyses. There are even methods for estimating the reliability of this mean series (e.g., average score intraclass correlations; McGraw & Wong, 1996). One warning I would offer, however, is that count data (which you seem to describe) is not always distributed normally. In the case of a highly skewed distribution, alternative methods for reliability will be necessary (e.g.,  a weighted chance-adjusted agreement index). 
If you have multiple raters provide categorical ratings for all items, the mean of these ratings will no longer be meaningful in most cases. However, you could still use a heuristic or voting procedure to aggregate ratings (e.g., use the modal category with some a priori tie-breaking procedure). Determining the reliability of the resulting aggregate will be tricky to estimate, but you could mirror the logic used for continuous ratings and compare the aggregate to all individual ratings and average the results.
If, however, you do not have multiple ratings for all items, then it would not make sense to average the results in just the subset for which you do. In this case, you should just pick one rater to use for each item (either randomly or using some explicit rationale).
In my work on nonverbal behavior, I have used all three of these approaches for different projects. When using expert raters on a very time-consuming task (Girard et al., 2014), we opted to use categorical ratings from individual raters for each item and use the most senior rater for those few items that were rated by multiple raters. When using non-expert raters on that same time-consuming task (McDuff, Girard, & el Kaliouby, in press), we opted to use a voting procedure to aggregate the categorical ratings of multiple raters for each item. And finally, when using raters on a quicker and highly inferential task (Ross, Girard, et al., 2016), we opted to average the continuous ratings of six different raters for each item.
References
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014). Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses. Image and Vision Computing, 32(10), 641–647.
McDuff, D., Girard, J. M., & El Kaliouby, R. (in press). Large-scale observational evidence of cross-cultural differences in facial behavior. Journal of Nonverbal Behavior.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
Ross, J. M., Girard, J. M., Wright, A. G. C., Beeney, J. E., Scott, L. N., Hallquist, M. N., … Pilkonis, P. A. (2016). Momentary patterns of covariation between specific affects and interpersonal behavior: Linking relationship science and personality assessment. Psychological Assessment.
