# What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few reliability coefficients to better understand them; now I'm trying to create an item-response model, per Andy W.'s suggestion.

First, a warning: I am not a qualified statistician in any discipline. It's been years since I ran a statistical analysis for any purpose, and at the time I wasn't terribly good at it. All the same, this is a lot of fun. I'm happy to run any analysis I can reasonably complete in R.

The dataset is available here. I truncated all but the last 20 columns. I also cleaned the data slightly. In some cases, "Unfriendly" was misspelled as "UIfriendly". In row 4352, two respondents, 19 and 20, had '0' in place of a response. Given the way the rest of the data set is organized, I assumed this is null data, thus I nulled out those two responses.

After reading this example of ordinal data modeling (specifically section 3.2) and this presentation about the 'ltm' package in R, I concluded my results were polytomous and I needed a graded response model. I first collected descriptive statistics of the data set. They suggested nothing surprising: "Neutral/unclear" was the most common response; no comment was universally friendly; some were universally unfriendly; respondents varied a fair bit in their frequency of neutral responses (see my other question for some measures of respondent agreement).

My sources then suggested calculating the data's nonparametric correlation coefficients, which I have done here (you'll want to toggle text wrapping), using Kendall's tau. If I'm reading these results correctly, I believe they suggest some association between respondents.

Finally, I ran the graded response models. One of my sources suggested fitting both a "constrained" and "unconstrained" model and comparing the two with an ANOVA test. Full results are linked to their respective models, below. I've pasted the summary lines here, as well as the results of the ANOVA:

Constrained GRM:

Model Summary:
log.Lik      AIC    BIC
-96657.64 193401.3 193696


Unconstrained GRM:

Model Summary:
log.Lik      AIC    BIC
-96141.05 192406.1 192831


ANOVA:

Likelihood Ratio Table
AIC    BIC   log.Lik     LRT df p.value
fit1 193401.3 193696 -96657.64
fit2 192406.1 192831 -96141.05 1033.18 19  <0.001


If I'm reading that ANOVA LRT correctly, it looks like the unconstrained model is a much better fit, so let's graph it. Everyone loves a graph. Out of the many I could post, I selected four: The combined Item Information Curves for the 20 respondents, and the Item Response Category Characteristic Curves for "Friendly" (category 1), "Neutral/unclear" (category 2), and "Unfriendly" (category 3). I'm happy to post any others that might be useful. They are, respectively:

Plot 1:

Plot 2:

Plot 3:

Plot 4:

I lack the training to interpret these results, so I'm hoping someone out there can. What did I measure and what, if any, conclusions can I draw?

-
You've analyzed the data as if it were from a completely crossed design, with each of 20 people rating all 7000 comments, but I can't find anything in the writeup that rules out a nested design, with each of the 7000 comments rated by a different set of 20 raters, or something somewhere in between, some sort of incomplete block design. How did you decide that the design was completely crossed? –  Ray Koopman Jul 16 '13 at 4:36
@RayKoopman: It was an arbitrary decision based on my reading of the writeup. It is not necessarily correct, and neither this analysis nor the experiment itself are especially rigorous. –  Christopher Jul 16 '13 at 17:11
Then we need to hear from the people who did the study just exactly what the design was, and if it was "somewhere in between" then we need to know which people rated which comments. –  Ray Koopman Jul 16 '13 at 17:29