If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He discusses a statistical analysis conducted on a sample of SE comments to evaluate their "friendliness" from an outside user's perspective. The comments were randomly sampled from StackOverflow and the content analysts were members of Amazon's Mechanical Turk community, a market for work that connects companies to workers who do small, short tasks for affordable fees.

Not so long ago, I was a graduate student in political science and one of the classes I took was Statistical Content Analysis. The class's final project, in fact its entire purpose, was to conduct a detailed analysis of the New York Times' war reporting, to test whether or not many assumptions Americans make about news coverage during wars were accurate (spoiler: evidence suggests they're not). The project was huge and quite fun, but by far its most painful section was the 'training and reliability testing phase', which occurred before we could conduct full analysis. It had two purposes (see page 9 of the linked paper for a detailed description, as well as references to intercoder reliability standards in the content analysis statistical literature):

  1. Confirm all coders, i.e., readers of the content, were trained on the same qualitative definitions. In Joel's analysis, this meant everyone would know exactly how the project defined "friendly" and "unfriendly."

  2. Confirm all coders interpreted these rules reliably, i.e. we sampled our sample, analyzed the subset, and then statistically demonstrated our pairwise correlations on qualitative evaluations were quite similar.

Reliability testing hurt because we had to do it three or four times. Until -1- was locked down and -2- showed high enough pairwise correlations, our results for the full analysis were suspect. They couldn't be demonstrated valid or invalid. Most importantly, we had to do pilot tests of reliability before the final sample set.

My question is this: Joel's statistical analysis lacked a pilot reliability test and didn't establish any operational definitions of "friendliness". Was the final data reliable enough to say anything about the statistical validity of his results?

For one perspective, consider this primer on the value of intercoder reliability and consistent operational definitions. From deeper in the same source, you can read about pilot reliability tests (item 5 in the list).

Per Andy W.'s suggestion in his answer, I'm attempting to calculate a variety of reliability statistics on the dataset, which is available here, using this command series in R (updated as I calculate new statistics).

Descriptive statistics are here

Percentage agreement (with tolerance = 0): 0.0143

Percentage agreement (with tolerance = 1): 11.8

Krippendorff's alpha: 0.1529467

I also attempted an item-response model for this data in another question.

  • 1
    The did publicly release the coding data so one could go and assess the reliability of the coders themselves if one wanted to. – Andy W Aug 2 '12 at 17:53
  • 3
    Re: #1 - It should be noted that this wasn't so much an exercise on if the comments were friendly or not, but more of an exercise on if the comments were perceived as friendly or not to an outside user. – Rachel Aug 2 '12 at 19:26
  • 3
    @Rachel I don't think that's right. If they were measuring how outsiders perceive comments on SO, they'd have needed quite a larger sample set than 20 people. – Christopher Aug 2 '12 at 19:52
  • 2
    It's the difference between concluding something about how outsiders perceive the comments, and concluding something about the comments themselves. In the first case, you'd need a much larger sample of people, and the conclusion would be "Outsiders think 2.3% of SO comments are unfriendly." In the second, it's "2.3% of SO comments are unfriendly." They're different conclusions, and I think the second one might not be possible to make, because we can't demonstrate the coders evaluate the comments similarly without a reliability test. – Christopher Aug 2 '12 at 19:56
  • 2
    @Christopher Friendliness is very subjective though. Depending on who you ask, the same comment can be viewed as both friendly and unfriendly. That is why I think its more important to get the point of view from a large number of random users instead of someone someone who has the exact same viewpoint as yourself. – Rachel Aug 2 '12 at 20:04
up vote 5 down vote accepted

Percentage agreement (with tolerance = 0): 0.0143

Percentage agreement (with tolerance = 1): 11.8

Krippendorff's alpha: 0.1529467

These agreement measures state that there is virtually no categorial agreement - each coder has his or her own internal cutoff point for judging comments as "friendly" or "unfriendly".

If we assume that the three categories are ordered, i.e.: Unfriendly < Neutral < Friendly, we can also calculate the intraclass correlation as another measure of agreement. On a random sample of 1000 comments, there is an ICC (2,1) of .28, and an ICC (2, k) of .88. That means, if you would only take one of the 20 raters, results would be very unreliable (.28), if you take the average of 20 raters, results are reliable (.88). Taking different combinations of three random raters, the averaged reliability is between .50 and .60, which still would be judged to be too low.

The average bivariate correlation between two coders is .34, which also is rather low.

If these agreement measures are seen as a quality measure of the coders (who actually should show good agreement), the answer is: they are not good coders and should be better trained. If this is seen as a measure of "how good is spontaneous agreement amongst random persons", the answer also is: not very high. As a benchmark, the average correlation for physical attractiveness ratings is around .47 - .71 [1]

[1] Langlois, J. H., Kalakanis, L., Rubenstein, A. J., Larson, A., Hallam, M., & Smoot, M. (2000). Maxims or myths of beauty? A meta-analytic and theoretical review. Psychological Bulletin, 126, 390–423. doi:10.1037/0033-2909.126.3.390

Reliability of scores is frequently interpreted in terms of Classical Test Theory. Here one has a true score, X, but what you observe at any particular outcome is not only the true score, but the true score with some error (i.e. Observed = X + error). In theory, by taking multiple observed measures of the same underlying test (making some assumptions about the distribution of the errors of those tests) one can then measure the unobserved true score.

Note here in this framework that you have to assume that your multiple observed measures are measuring the same underlying test. Poor reliability of test items is then frequently taken as evidence that the observed measures are not measuring the same underlying test. This is just a convention of the field though, poor reliability, in and of itself, does not prove (in any statistical sense) the items are not measuring the same construct. So it could be argued that by taking many observed measures, even with very unreliable tests, one could come about a reliable measure of the true score.

It also stands to be mentioned that classical test theory isn't necessarily the only way to interpret such tests, and many scholors would argue that the concept of latent variables and item-response theory is always more appropriate than classical test theory.

Also a similar implicit assumption in classical test theory is when people say reliabilities are too high. It doesn't say anything about the validity of whether particular item(s) measure some underlying test, but that when reliabilities are too high researchers take it as evidence that errors between the tests are not independent.

I'm not quite sure why your so vehement about not going in and calculating the reliabilities yourself. Why could one not do this and subsequently interpret the analysis in light of this extra information?

  • So first let me point out that I'm not a grad student doing stats anymore for a good reason: it wasn't quite my forte. I might be misremembering the methodology. All the same, I think you and I might be talking about different measures of reliability, or at the least there is research to suggest measuring intercoder reliability before the final analysis is conducted matters for validity. I've edited the question to include one source I found on the web, which cites considerably more research on the subject. – Christopher Aug 2 '12 at 19:15
  • It is a different context (reliability of dichotomous test items instead of some continuous outcome), but the logic is functionally the same. Hence why I did not mention any specific measure of reliability (there are many). Your quote does not insinuate anything about before the final analysis, so I'm not quite sure where that notion comes from. – Andy W Aug 2 '12 at 19:54
  • Ah ha. You are correct, it is not quite a requirement. Reading further into that link I posted, it looks like this pilot tests are considered a methodological best practice (search for pilot test in it). – Christopher Aug 2 '12 at 19:59
  • I've changed my question to accommodate the new information. Thank you for the help correcting my error. – Christopher Aug 2 '12 at 20:07
  • 2
    Other question is up. – Christopher Aug 3 '12 at 22:08

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.