Is Joel Spolsky's "Hunting of the Snark" post valid statistical content analysis?

Question

If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He discusses a statistical analysis conducted on a sample of SE comments to evaluate their "friendliness" from an outside user's perspective. The comments were randomly sampled from StackOverflow and the content analysts were members of Amazon's Mechanical Turk community, a market for work that connects companies to workers who do small, short tasks for affordable fees.

Not so long ago, I was a graduate student in political science and one of the classes I took was Statistical Content Analysis. The class's final project, in fact its entire purpose, was to conduct a detailed analysis of the New York Times' war reporting, to test whether or not many assumptions Americans make about news coverage during wars were accurate (spoiler: evidence suggests they're not). The project was huge and quite fun, but by far its most painful section was the 'training and reliability testing phase', which occurred before we could conduct full analysis. It had two purposes (see page 9 of the linked paper for a detailed description, as well as references to intercoder reliability standards in the content analysis statistical literature):

1. Confirm all coders, i.e., readers of the content, were trained on the same qualitative definitions. In Joel's analysis, this meant everyone would know exactly how the project defined "friendly" and "unfriendly."

2. Confirm all coders interpreted these rules reliably, i.e. we sampled our sample, analyzed the subset, and then statistically demonstrated our pairwise correlations on qualitative evaluations were quite similar.

Reliability testing hurt because we had to do it three or four times. Until -1- was locked down and -2- showed high enough pairwise correlations, our results for the full analysis were suspect. They couldn't be demonstrated valid or invalid. Most importantly, we had to do pilot tests of reliability before the final sample set.

My question is this: Joel's statistical analysis lacked a pilot reliability test and didn't establish any operational definitions of "friendliness". Was the final data reliable enough to say anything about the statistical validity of his results?

For one perspective, consider this primer on the value of intercoder reliability and consistent operational definitions. From deeper in the same source, you can read about pilot reliability tests (item 5 in the list).

Per Andy W.'s suggestion in his answer, I'm attempting to calculate a variety of reliability statistics on the dataset, which is available here, using this command series in R (updated as I calculate new statistics).

Descriptive statistics are here

Percentage agreement (with tolerance = 0): 0.0143

Percentage agreement (with tolerance = 1): 11.8

Krippendorff's alpha: 0.1529467

I also attempted an item-response model for this data in another question.

Answer 1

Percentage agreement (with tolerance = 0): 0.0143

Percentage agreement (with tolerance = 1): 11.8

Krippendorff's alpha: 0.1529467

These agreement measures state that there is virtually no categorial agreement - each coder has his or her own internal cutoff point for judging comments as "friendly" or "unfriendly".

If we assume that the three categories are ordered, i.e.: Unfriendly < Neutral < Friendly, we can also calculate the intraclass correlation as another measure of agreement. On a random sample of 1000 comments, there is an ICC (2,1) of .28, and an ICC (2, k) of .88. That means, if you would only take one of the 20 raters, results would be very unreliable (.28), if you take the average of 20 raters, results are reliable (.88). Taking different combinations of three random raters, the averaged reliability is between .50 and .60, which still would be judged to be too low.

The average bivariate correlation between two coders is .34, which also is rather low.

If these agreement measures are seen as a quality measure of the coders (who actually should show good agreement), the answer is: they are not good coders and should be better trained. If this is seen as a measure of "how good is spontaneous agreement amongst random persons", the answer also is: not very high. As a benchmark, the average correlation for physical attractiveness ratings is around .47 - .71 [1]

[1] Langlois, J. H., Kalakanis, L., Rubenstein, A. J., Larson, A., Hallam, M., & Smoot, M. (2000). Maxims or myths of beauty? A meta-analytic and theoretical review. Psychological Bulletin, 126, 390–423. doi:10.1037/0033-2909.126.3.390

Answer 2

Reliability of scores is frequently interpreted in terms of Classical Test Theory. Here one has a true score, X, but what you observe at any particular outcome is not only the true score, but the true score with some error (i.e. Observed = X + error). In theory, by taking multiple observed measures of the same underlying test (making some assumptions about the distribution of the errors of those tests) one can then measure the unobserved true score.

Note here in this framework that you have to assume that your multiple observed measures are measuring the same underlying test. Poor reliability of test items is then frequently taken as evidence that the observed measures are not measuring the same underlying test. This is just a convention of the field though, poor reliability, in and of itself, does not prove (in any statistical sense) the items are not measuring the same construct. So it could be argued that by taking many observed measures, even with very unreliable tests, one could come about a reliable measure of the true score.

It also stands to be mentioned that classical test theory isn't necessarily the only way to interpret such tests, and many scholors would argue that the concept of latent variables and item-response theory is always more appropriate than classical test theory.

---

Also a similar implicit assumption in classical test theory is when people say reliabilities are too high. It doesn't say anything about the validity of whether particular item(s) measure some underlying test, but that when reliabilities are too high researchers take it as evidence that errors between the tests are not independent.

I'm not quite sure why you are so vehement about not going in and calculating the reliabilities yourself. Why could one not do this and subsequently interpret the analysis in light of this extra information?