Skip to main content
deleted 11 characters in body; edited tags
Source Link
kjetil b halvorsen
  • 82.8k
  • 32
  • 201
  • 663

I apologize for the vague title, but it's quite hard for to me see how I can succinctly explain what I need.

So, for example there is a sentence that was coded 10 times by 10 different coders. That is, one sentence is 10 different observations. The scale is ordinal from 1 to 5. Now, I have 100 sentences (100 * 10 = 1000 observations), each of which have been coded by different people.

What I want to know is whether there is some metric that would allow me to check how consistent each sentence is coded.

For example: if Sentence 1 is coded as 1, 1, 1, 2, 1, 2, 2, 1, 1, 1 and Sentence 2 is coded as 1, 5, 2, 1, 3, 3, 1, 4, 5, 2 one could say that the first sentence is more consistent in how it's coded.

I can't use any of the interceder reliability metrics, since the coders are not the same for every sentence, therefore it doesn't make any sense to see whether they agree across sentence. I just need to check whether the coding for one sentence (10 observations) is consistent.

Thanks.

I apologize for the vague title, but it's quite hard for to me see how I can succinctly explain what I need.

So, for example there is a sentence that was coded 10 times by 10 different coders. That is, one sentence is 10 different observations. The scale is ordinal from 1 to 5. Now, I have 100 sentences (100 * 10 = 1000 observations), each of which have been coded by different people.

What I want to know is whether there is some metric that would allow me to check how consistent each sentence is coded.

For example: if Sentence 1 is coded as 1, 1, 1, 2, 1, 2, 2, 1, 1, 1 and Sentence 2 is coded as 1, 5, 2, 1, 3, 3, 1, 4, 5, 2 one could say that the first sentence is more consistent in how it's coded.

I can't use any of the interceder reliability metrics, since the coders are not the same for every sentence, therefore it doesn't make any sense to see whether they agree across sentence. I just need to check whether the coding for one sentence (10 observations) is consistent.

Thanks.

I apologize for the vague title, but it's quite hard for to me see how I can succinctly explain what I need.

So, for example there is a sentence that was coded 10 times by 10 different coders. That is, one sentence is 10 different observations. The scale is ordinal from 1 to 5. Now, I have 100 sentences (100 * 10 = 1000 observations), each of which have been coded by different people.

What I want to know is whether there is some metric that would allow me to check how consistent each sentence is coded.

For example: if Sentence 1 is coded as 1, 1, 1, 2, 1, 2, 2, 1, 1, 1 and Sentence 2 is coded as 1, 5, 2, 1, 3, 3, 1, 4, 5, 2 one could say that the first sentence is more consistent in how it's coded.

I can't use any of the interceder reliability metrics, since the coders are not the same for every sentence, therefore it doesn't make any sense to see whether they agree across sentence. I just need to check whether the coding for one sentence (10 observations) is consistent.

Source Link

Intercoder consistency for a single observation

I apologize for the vague title, but it's quite hard for to me see how I can succinctly explain what I need.

So, for example there is a sentence that was coded 10 times by 10 different coders. That is, one sentence is 10 different observations. The scale is ordinal from 1 to 5. Now, I have 100 sentences (100 * 10 = 1000 observations), each of which have been coded by different people.

What I want to know is whether there is some metric that would allow me to check how consistent each sentence is coded.

For example: if Sentence 1 is coded as 1, 1, 1, 2, 1, 2, 2, 1, 1, 1 and Sentence 2 is coded as 1, 5, 2, 1, 3, 3, 1, 4, 5, 2 one could say that the first sentence is more consistent in how it's coded.

I can't use any of the interceder reliability metrics, since the coders are not the same for every sentence, therefore it doesn't make any sense to see whether they agree across sentence. I just need to check whether the coding for one sentence (10 observations) is consistent.

Thanks.