0
$\begingroup$

I apologize for the vague title, but it's quite hard for to me see how I can succinctly explain what I need.

So, for example there is a sentence that was coded 10 times by 10 different coders. That is, one sentence is 10 different observations. The scale is ordinal from 1 to 5. Now, I have 100 sentences (100 * 10 = 1000 observations), each of which have been coded by different people.

What I want to know is whether there is some metric that would allow me to check how consistent each sentence is coded.

For example: if Sentence 1 is coded as 1, 1, 1, 2, 1, 2, 2, 1, 1, 1 and Sentence 2 is coded as 1, 5, 2, 1, 3, 3, 1, 4, 5, 2 one could say that the first sentence is more consistent in how it's coded.

I can't use any of the interceder reliability metrics, since the coders are not the same for every sentence, therefore it doesn't make any sense to see whether they agree across sentence. I just need to check whether the coding for one sentence (10 observations) is consistent.

Improve this question
$\endgroup$
2
  • $\begingroup$ This question is not clear. $\endgroup$
    – Michael R. Chernick
    Commented Apr 27, 2017 at 16:02
  • $\begingroup$ You could look into kripperdorffs alpha, which is a coefficient of agreement which can use ordinal data, more than two coders and some missing data. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 13, 2017 at 20:34

2 Answers 2

Reset to default
0
$\begingroup$

If the coding is consistent then all the coders will give the same value.

If one coder is different from another coder then they are not consistent.

So each sentence is either consistent or not consistent.

There doesn't seem to be much more you can do.

Improve this answer
$\endgroup$
0
$\begingroup$

Cees van der Eijk proposed a very nifty perceptual agreement coefficient for ordinal variables (such as your 5-point scale) that can be applied to small samples such as yours (10 raters, with one rating for each). Calculating coefficient A for each of your sentences is straightfoward, especially through its implementation in Stata where you can import your data with each column representing a coder and each row representing a sentence.

Van der Eijk, C. (2001). Measuring agreement in ordered rating scales. Quality and Quantity, 35(3), 325-341. https://doi.org/10.1023/A:1010374114305

Agreement A package for Stata: https://ideas.repec.org/c/boc/bocode/s457287.html

Agreement A package for R: https://rdrr.io/rforge/agrmt/man/agreement.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.