In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few reliability coefficients to better understand them; now I'm trying to create an item-response model, per Andy W.'s suggestion.
First, a warning: I am not a qualified statistician in any discipline. It's been years since I ran a statistical analysis for any purpose, and at the time I wasn't terribly good at it. All the same, this is a lot of fun. I'm happy to run any analysis I can reasonably complete in R.
The dataset is available here. I truncated all but the last 20 columns. I also cleaned the data slightly. In some cases, "Unfriendly" was misspelled as "UIfriendly". In row 4352, two respondents, 19 and 20, had '0' in place of a response. Given the way the rest of the data set is organized, I assumed this is null data, thus I nulled out those two responses.
After reading this example of ordinal data modeling (specifically section 3.2) and this presentation about the 'ltm' package in R, I concluded my results were polytomous and I needed a graded response model. I first collected descriptive statistics of the data set. They suggested nothing surprising: "Neutral/unclear" was the most common response; no comment was universally friendly; some were universally unfriendly; respondents varied a fair bit in their frequency of neutral responses (see my other question for some measures of respondent agreement).
My sources then suggested calculating the data's nonparametric correlation coefficients, which I have done here (you'll want to toggle text wrapping), using Kendall's tau. If I'm reading these results correctly, I believe they suggest some association between respondents.
Finally, I ran the graded response models. One of my sources suggested fitting both a "constrained" and "unconstrained" model and comparing the two with an ANOVA test. Full results are linked to their respective models, below. I've pasted the summary lines here, as well as the results of the ANOVA:
Model Summary: log.Lik AIC BIC -96657.64 193401.3 193696
Model Summary: log.Lik AIC BIC -96141.05 192406.1 192831
Likelihood Ratio Table AIC BIC log.Lik LRT df p.value fit1 193401.3 193696 -96657.64 fit2 192406.1 192831 -96141.05 1033.18 19 <0.001
If I'm reading that ANOVA LRT correctly, it looks like the unconstrained model is a much better fit, so let's graph it. Everyone loves a graph. Out of the many I could post, I selected four: The combined Item Information Curves for the 20 respondents, and the Item Response Category Characteristic Curves for "Friendly" (category 1), "Neutral/unclear" (category 2), and "Unfriendly" (category 3). I'm happy to post any others that might be useful. They are, respectively:
I lack the training to interpret these results, so I'm hoping someone out there can. What did I measure and what, if any, conclusions can I draw?