2
$\begingroup$

I have essay scores from one experiment. I have two raters and each rater separately graded the essays by using rubrics. The rubrics consist of 5 criteria, rating scales (1-5) and "descriptions"(clearly definition of a certain level on that criterion) in each cell.

I have three questions: - Are scores of each criterion regarded as ordinal or interval data? Which statistical method is more appropriate for rater agreements? - Should I calculate rater agreements for the scores per criterion or the total score (adding up scores of all criteria)? - Can I run parametric analysis to compare the essay scores of different groups of students?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The scores are regarded as ordinal. You mentioned the range is 1-5. By establishing a lower and upper limit, the difference between two numbers that have a ratings difference of 1 are not equal (for example the difference between an essay rated 4 and an essay rated 5 wouldn't necessarily equal the difference between an essay rated 3 and an essay rated 4.

I wrote a python script that looks at each of the score differences among the two graders for each of the criteria compared to the average differences among all essay's grader's scores (with each essay having two sets of scores of size 5, each signifying the 5 criteria scores the grader gave the essay). It's got some issues including:

  1. Performs parametric analysis.
  2. Assumes that the distribution of score differences is normal.

but it gives a good estimate of the likeness of the two grades each essay's graders give of each of the criteria with respect to how alike other essay's scores are.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Many thanks! But I thought ordinal data cannot be analyzed with non-parametric methods... $\endgroup$
    – ahs
    Oct 2, 2015 at 12:42
  • $\begingroup$ It's actually the other way around: ordinal data should only be analyzed with non-parametric methods. But you're right in that I used the wrong type of analysis for ordinal data (I used parametric methods instead of non-parametric methods). I justed wanted to see if parametric analysis would work for you if examining the difference between two ordinal data points (the difference between two grades for a certain criteria awarded to an essay by two different graders). $\endgroup$
    – Antoine Dahan
    Oct 2, 2015 at 13:33
0
$\begingroup$

For question 1: the ratings are ordinal. See the wikipedia entry here.

For question 2: I think the question begs some clarification.

For question 3: it seems unlikely that parametric analysis is the way to go. I'd suggest a non-parametric approach, depending on the nature of your hypotheses.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.