Calculation for inter-rater reliability where raters don't overlap and different number per candidate?

I want to calculate the degree to which the gymnastics judges agree on balance beam scores, i.e., "inter-rater reliability". However, not all judges judge the same candidates, and the number of judges per candidate also varies. There are around 30 judges making roughly 1500 observations.

The data looks like this:

Can you please tell me how to do this statistically, perhaps using Cronbach's alpha?

STATA set-up advice would help, as well.

• Questions solely about how software works are off topic here, but you may have a real statistical question buried here. You may want to edit your question to clarify the underlying statistical issue. You may find that when you understand the statistical concepts involved, the software-specific elements are self-evident or at least easy to get from the documentation. – gung Nov 10 '15 at 1:31

Here's an example using the kappa-statistic measure of interrater agreement. Before proceeding, we will need to reshape the data so that each row is a gymnast but each score variable corresponds to single judge.

clear
input byte Gymnast str9 Judge double Score
1 Smith 5.5
1 Bartlet 6
1 Baily 8
2 Smith 10
2 Patterson 9.5
3 Baily 8
3 Patterson 7
3 Smith 7.5
4 Bartlet 7.5
end
rename Score Score_
reshape wide Score_, i(Gymnast) j(Judge, string)
kap Score_*


The combined kappa is -0.1912, which would be considered poor. Stata recommends the following RoTs for summarizing agreement:

below 0.0 Poor
0.00 – 0.20 Slight
0.21 – 0.40 Fair
0.41 – 0.60 Moderate
0.61 – 0.80 Substantial
0.81 – 1.00 Almost perfect

• If I do that, I get: "Judge not unique within Gymnast; there are multiple observations at the same Judge within Gymnast." I'm not sure what that means except I think it's because not all Gymnasts have the same set of Judges.... – EconoQ Nov 12 '15 at 0:14
• It's probably not that since in my example the set of judges varies across gymnasts and it works. Before you reshape, try duplicates tag Gymnast Judge, gen(dups) and tab dups. If dups is ever above zero, you have the same gymnast judged twice by the same judge. This could happen if the gymnast competes in two separate competitions. If that is the case, you have panel data, which is harder to model and will require some additional assumptions before you can proceed. – Dimitriy V. Masterov Nov 12 '15 at 0:39
• Maybe I have 2 duplicates? In which case, perhaps I can just drop those points? Dups Freq Percent Cum1 0 1140 99.82 99.82 1 2 0.18 100 Total 1142 100.00 – EconoQ Nov 12 '15 at 0:52
• Looks like you have 2 duplicates. It might well be a data entry error, so take a look at the scores. They might be the same, so you can just drop one of the rows. If not, you have to use your judgment. – Dimitriy V. Masterov Nov 12 '15 at 1:21
• I went ahead and dropped the row. But now I get: "kap Score* matsize too small. You have attempted to create a matrix with too many rows or columns or attempted to fit a model with too many variables. You need to increase matsize; it is currently 800...." – EconoQ Nov 12 '15 at 15:34