What am I measuring when I apply a graded response model to the "Hunting of the Snark" dataset?

Question

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few reliability coefficients to better understand them; now I'm trying to create an item-response model, per Andy W.'s suggestion.

First, a warning: I am not a qualified statistician in any discipline. It's been years since I ran a statistical analysis for any purpose, and at the time I wasn't terribly good at it. All the same, this is a lot of fun. I'm happy to run any analysis I can reasonably complete in R.

The dataset is available here. I truncated all but the last 20 columns. I also cleaned the data slightly. In some cases, "Unfriendly" was misspelled as "UIfriendly". In row 4352, two respondents, 19 and 20, had '0' in place of a response. Given the way the rest of the data set is organized, I assumed this is null data, thus I nulled out those two responses.

After reading this example of ordinal data modeling (specifically section 3.2) and this presentation about the 'ltm' package in R, I concluded my results were polytomous and I needed a graded response model. I first collected descriptive statistics of the data set. They suggested nothing surprising: "Neutral/unclear" was the most common response; no comment was universally friendly; some were universally unfriendly; respondents varied a fair bit in their frequency of neutral responses (see my other question for some measures of respondent agreement).

My sources then suggested calculating the data's nonparametric correlation coefficients, which I have done here (you'll want to toggle text wrapping), using Kendall's tau. If I'm reading these results correctly, I believe they suggest some association between respondents.

Finally, I ran the graded response models. One of my sources suggested fitting both a "constrained" and "unconstrained" model and comparing the two with an ANOVA test. Full results are linked to their respective models, below. I've pasted the summary lines here, as well as the results of the ANOVA:

Constrained GRM:

Model Summary:
    log.Lik      AIC    BIC
    -96657.64 193401.3 193696

Unconstrained GRM:

Model Summary:
    log.Lik      AIC    BIC
    -96141.05 192406.1 192831

ANOVA:

Likelihood Ratio Table
          AIC    BIC   log.Lik     LRT df p.value
fit1 193401.3 193696 -96657.64                   
fit2 192406.1 192831 -96141.05 1033.18 19  <0.001

If I'm reading that ANOVA LRT correctly, it looks like the unconstrained model is a much better fit, so let's graph it. Everyone loves a graph. Out of the many I could post, I selected four: The combined Item Information Curves for the 20 respondents, and the Item Response Category Characteristic Curves for "Friendly" (category 1), "Neutral/unclear" (category 2), and "Unfriendly" (category 3). I'm happy to post any others that might be useful. They are, respectively:

Plot 1:

Plot 2:

Plot 3:

Plot 4:

I lack the training to interpret these results, so I'm hoping someone out there can. What did I measure and what, if any, conclusions can I draw?

Answer 1

The analyses you did are somewhat unorthodox. Using the GRM you usually want to locate each comment on a latent unidimensional trait from unfriendly to friendly. Therefore, one would use several polytomous items which are indicators of the latent trait. In your analysis, you treated the different raters as "items".

Plot 1 gives you the item (i.e., rater) information curve: the higher the curve, the better the reliability in the respective region of the trait. That means, the reddish raters are good in discriminating friendly and unfriendly comments, but not so good in discriminating neutral comments. The greenish raters generally are worse in discriminating comments (their curves are lower overall), especially friendly comments.

The next three plots (these are ICC plots) give you basically the same information as the first plot: the steeper the curve in the ICC plot, the more information is obtained.

It is interesting that there seem to be two types of raters: lenient and demanding. The greenish raters are rather lenient: their boundaries between friendly and neutral, and between neutral and unfriendly are moved to the right (as you can see in Plots 2-4). That means, when a comment moves the latent trait from friendly to unfriendly, they switch the categories later to neutral and to unfriendly.