I have essay scores from one experiment. I have two raters and each rater separately graded the essays by using rubrics. The rubrics consist of 5 criteria, rating scales (1-5) and "descriptions"(clearly definition of a certain level on that criterion) in each cell.
I have three questions:
- Are scores of each criterion regarded as ordinal or interval data? Which statistical method is more appropriate for rater agreements?
- Should I calculate rater agreements for the scores per criterion or the total score (adding up scores of all criteria)?
- Can I run parametric analysis to compare the essay scores of different groups of students?
The scores are regarded as ordinal. You mentioned the range is 1-5. By establishing a lower and upper limit, the difference between two numbers that have a ratings difference of 1 are not equal (for example the difference between an essay rated 4 and an essay rated 5 wouldn't necessarily equal the difference between an essay rated 3 and an essay rated 4.
I wrote a python script that looks at each of the score differences among the two graders for each of the criteria compared to the average differences among all essay's grader's scores (with each essay having two sets of scores of size 5, each signifying the 5 criteria scores the grader gave the essay). It's got some issues including:
Performs parametric analysis.
Assumes that the distribution of score differences is normal.
but it gives a good estimate of the likeness of the two grades each essay's graders give of each of the criteria with respect to how alike other essay's scores are.