I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters.
What I do is following: for each feature I organize data such that I get the matrix of 9 columns as categories (corresponding to each level of the rating scale) and 16 rows for samples that were rated. Cells i,j of the matrix is a count of subject who applied j-th rating to i-th sample. Then I apply Fleiss kappa to the matrix in order to get inter-rater agreement.
My questions are: 1.am I using kappa statistic correctly or did I get it wrong? 2.I have agreement barely above chance level (Kappa=0.04) while average of all inter-subject correlations is 0.4 which should be significantly higher than chance level (Hence the question 1).