-1
$\begingroup$

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters.

What I do is following: for each feature I organize data such that I get the matrix of 9 columns as categories (corresponding to each level of the rating scale) and 16 rows for samples that were rated. Cells i,j of the matrix is a count of subject who applied j-th rating to i-th sample. Then I apply Fleiss kappa to the matrix in order to get inter-rater agreement.

My questions are: 1.am I using kappa statistic correctly or did I get it wrong? 2.I have agreement barely above chance level (Kappa=0.04) while average of all inter-subject correlations is 0.4 which should be significantly higher than chance level (Hence the question 1).

$\endgroup$
0
$\begingroup$

Without knowing which implementation of Fleiss kappa you are using, it's impossible to say if you are using it correctly. Some functions want the data to be in the format you described, while others want the data to be in other formats. If you'd like to check your result again, you can try my formulation of Fleiss' kappa here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.