Six raters independently rated 25 people by identifying strengths and weaknesses: choosing among 35 traits (15 positive traits and 20 negative traits). The rater instructions said to identify a maximum of the 3 greatest strengths and the 3 greatest weaknesses for each person. On average, a person received about 3 strength or weakness ratings from a rater (less than the 6 expected based on the rater instructions).
How might I go about describing the degree of rater agreement? (The options that come to mind are % agreement and some index of inter-rater reliability.)
My primary interest is in the person level agreement, since the people involved are interested in knowing whether the raters are giving them valuable information. The overall level of agreement is also of interest.
Update: A Possible Approach:
This dilemma seems to have 2 separate aspects: (a) developing a measure of the level of rater agreement, and (b) developing a way to compare the observed level of rater agreement to chance. Here is an approach that I describe in hope of getting constructive feedback.
(a) Rater Agreement Ratio
Define rater agreement as the ratio of the total number of attributes selected by the raters (counting each time an attribute is chosen, whether or not the attribute is chosen by multiple raters) to the total number of unique attributes selected (counting each chosen attribute once, no matter how many raters chose that attribute)
Since there are 6 raters, if all raters chose the same attributes, the ratio would be 6.0. For example, if all 6 raters chose attributes 1 and 3, the rater agreement ratio would be 12/2=6. If all raters chose different attributes, the ratio would be 1.0. For example, if the 6 raters chose 18 different attributes, the rater agreement ratio would be 18/18=1.0. So this measure could range from 1) no agreemnt0 to 6 (perect agreement).
(b) Distribution of rater agreement ratio based on chance
Calculate the extent of agreement that would be seen based on chance using a Monte Carlo study, as follows.
For each rater, tally the attributes chosen to get a frequency distribution of the positive and negative attributes chosen.
Use these frequency distributions to calculate level of agreement that would be observed based on chance, as follows. If rater A chose 3 positive and 2 negative attributes for person 1, select 3 positive and 2 negative attributes from these two pools of attributes for rater A: (1) all the positive attributes chosen by rater A for all people, and (2) all the negative attributes chosen by rater A for all people, in both cases allowing for frequency of attribute choices. Repeat this for Raters B through F. Then calculate the rater agreement ratio due to chance. Repeat this multiple times (Monte Carlo approach) to get the degree of agreement expected based on chance and the probability distribution of the rater agreement ratio. Compare this with the level of agreement actually observed.
What do you think?