How to calculate correlations of matching pairs I have very basic knowledge of statistics and would appreciate some help identifying an appropriate method to calculate the proportion of outcomes that agree between pairs of data and whether it is significant.
The dataset consists of 100+ cases that have been reviewed 6 reviewers, but not all reviewers will review all cases so there are gaps in the table. There are three outcomes that I would like to compare that are coded '1,2,3' to identify the proportion of outcomes that match between pairs of reviewers. Where a case has not been reviewed by both reviewers I do not want this to count as a match. 
 A: For a particular pair of reviewers, you can compare their interrater agreement by looking at a statistical measure called Cohen's Kappa. This measure is meant to generalize the idea of percent agreement by accounting for the overall distribution of classifications. For instance, in coding mammographic screens as being positive or negative for cancer, most screens will come back negative, and this may make radiologists seem as though their agreement is very high until you use Cohen's Kappa instead.
Roughly, Cohen's Kappa is calculated using the following formula:
$$\kappa = \frac{Pr(a) - \bar{E}}{1-\bar{E}}$$
Where $a$ is the event of chance agreement and $\bar{E}$ is the hypothetical probability of a chance agreement (note $\bar{E}$ is not an event, hence there is no $Pr(e)$ as the Wiki article suggests).
For your analysis there are many ways that this can be applied. If the 6 raters themselves are inherently interesting, you may calculate the pairwise 6-choose-2 comparisons between raters and qualitatively examine who tends to agree with whom. Otherwise, you can define $a$ to be a unanimous agreement among all 6 raters using complete cases and defining $e$ according to the empirical probability that all 6 agree (presumably low). You can similarly calculate such values among all available cases, recalculating $\bar{E}$ for each row in the sample as a function of which raters applied ratings. 
It's also very important to verify that raters did not avoid rating due to the actual value that the case took, e.g. they found it too hard, or too insultingly easy to say. This is a big problem that is often overlooked in such analyses.
