In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few reliability coefficients to better understand them; now I'm trying to create an item-response model, per Andy W.'s suggestion.

First, a warning: I am not a qualified statistician in any discipline. It's been years since I ran a statistical analysis for any purpose, and at the time I wasn't terribly good at it. All the same, this is a lot of fun. I'm happy to run any analysis I can reasonably complete in R.

The dataset is available here. I truncated all but the last 20 columns. I also cleaned the data slightly. In some cases, "Unfriendly" was misspelled as "UIfriendly". In row 4352, two respondents, 19 and 20, had '0' in place of a response. Given the way the rest of the data set is organized, I assumed this is null data, thus I nulled out those two responses.

After reading this example of ordinal data modeling (specifically section 3.2) and this presentation about the 'ltm' package in R, I concluded my results were polytomous and I needed a graded response model. I first collected descriptive statistics of the data set. They suggested nothing surprising: "Neutral/unclear" was the most common response; no comment was universally friendly; some were universally unfriendly; respondents varied a fair bit in their frequency of neutral responses (see my other question for some measures of respondent agreement).

My sources then suggested calculating the data's nonparametric correlation coefficients, which I have done here (you'll want to toggle text wrapping), using Kendall's tau. If I'm reading these results correctly, I believe they suggest some association between respondents.

Finally, I ran the graded response models. One of my sources suggested fitting both a "constrained" and "unconstrained" model and comparing the two with an ANOVA test. Full results are linked to their respective models, below. I've pasted the summary lines here, as well as the results of the ANOVA:

Constrained GRM:

Model Summary:
    log.Lik      AIC    BIC
    -96657.64 193401.3 193696

Unconstrained GRM:

Model Summary:
    log.Lik      AIC    BIC
    -96141.05 192406.1 192831


Likelihood Ratio Table
          AIC    BIC   log.Lik     LRT df p.value
fit1 193401.3 193696 -96657.64                   
fit2 192406.1 192831 -96141.05 1033.18 19  <0.001

If I'm reading that ANOVA LRT correctly, it looks like the unconstrained model is a much better fit, so let's graph it. Everyone loves a graph. Out of the many I could post, I selected four: The combined Item Information Curves for the 20 respondents, and the Item Response Category Characteristic Curves for "Friendly" (category 1), "Neutral/unclear" (category 2), and "Unfriendly" (category 3). I'm happy to post any others that might be useful. They are, respectively:

Plot 1: Combined IIC

Plot 2: Category 1 -- "Friendly"

Plot 3: Category 2 -- "Neutral/unclear"

Plot 4: Category 3 -- "Unfriendly"

I lack the training to interpret these results, so I'm hoping someone out there can. What did I measure and what, if any, conclusions can I draw?

  • $\begingroup$ You've analyzed the data as if it were from a completely crossed design, with each of 20 people rating all 7000 comments, but I can't find anything in the writeup that rules out a nested design, with each of the 7000 comments rated by a different set of 20 raters, or something somewhere in between, some sort of incomplete block design. How did you decide that the design was completely crossed? $\endgroup$ Commented Jul 16, 2013 at 4:36
  • $\begingroup$ @RayKoopman: It was an arbitrary decision based on my reading of the writeup. It is not necessarily correct, and neither this analysis nor the experiment itself are especially rigorous. $\endgroup$ Commented Jul 16, 2013 at 17:11
  • $\begingroup$ Then we need to hear from the people who did the study just exactly what the design was, and if it was "somewhere in between" then we need to know which people rated which comments. $\endgroup$ Commented Jul 16, 2013 at 17:29

1 Answer 1


The analyses you did are somewhat unorthodox. Using the GRM you usually want to locate each comment on a latent unidimensional trait from unfriendly to friendly. Therefore, one would use several polytomous items which are indicators of the latent trait. In your analysis, you treated the different raters as "items".

Plot 1 gives you the item (i.e., rater) information curve: the higher the curve, the better the reliability in the respective region of the trait. That means, the reddish raters are good in discriminating friendly and unfriendly comments, but not so good in discriminating neutral comments. The greenish raters generally are worse in discriminating comments (their curves are lower overall), especially friendly comments.

The next three plots (these are ICC plots) give you basically the same information as the first plot: the steeper the curve in the ICC plot, the more information is obtained.

It is interesting that there seem to be two types of raters: lenient and demanding. The greenish raters are rather lenient: their boundaries between friendly and neutral, and between neutral and unfriendly are moved to the right (as you can see in Plots 2-4). That means, when a comment moves the latent trait from friendly to unfriendly, they switch the categories later to neutral and to unfriendly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.