If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He discusses a statistical analysis conducted on a sample of SE comments to evaluate their "friendliness" from an outside user's perspective. The comments were randomly sampled from StackOverflow and the content analysts were members of Amazon's Mechanical Turk community, a market for work that connects companies to workers who do small, short tasks for affordable fees.
Not so long ago, I was a graduate student in political science and one of the classes I took was Statistical Content Analysis. The class's final project, in fact its entire purpose, was to conduct a detailed analysis of the New York Times' war reporting, to test whether or not many assumptions Americans make about news coverage during wars were accurate (spoiler: evidence suggests they're not). The project was huge and quite fun, but by far its most painful section was the 'training and reliability testing phase', which occurred before we could conduct full analysis. It had two purposes (see page 9 of the linked paper for a detailed description, as well as references to intercoder reliability standards in the content analysis statistical literature):
Confirm all coders, i.e., readers of the content, were trained on the same qualitative definitions. In Joel's analysis, this meant everyone would know exactly how the project defined "friendly" and "unfriendly."
Confirm all coders interpreted these rules reliably, i.e. we sampled our sample, analyzed the subset, and then statistically demonstrated our pairwise correlations on qualitative evaluations were quite similar.
Reliability testing hurt because we had to do it three or four times. Until -1- was locked down and -2- showed high enough pairwise correlations, our results for the full analysis were suspect. They couldn't be demonstrated valid or invalid. Most importantly, we had to do pilot tests of reliability before the final sample set.
My question is this: Joel's statistical analysis lacked a pilot reliability test and didn't establish any operational definitions of "friendliness". Was the final data reliable enough to say anything about the statistical validity of his results?
For one perspective, consider this primer on the value of intercoder reliability and consistent operational definitions. From deeper in the same source, you can read about pilot reliability tests (item 5 in the list).
Per Andy W.'s suggestion in his answer, I'm attempting to calculate a variety of reliability statistics on the dataset, which is available here, using this command series in R (updated as I calculate new statistics).
Percentage agreement (with tolerance = 0): 0.0143
Percentage agreement (with tolerance = 1): 11.8
Krippendorff's alpha: 0.1529467
I also attempted an item-response model for this data in another question.