A friend of mine took a document and broke it up into parts, then asked 5 subject matter experts to classify each part into nominal category A, B, C, D, or E. (I'm not sure yet, but D may be "All of the Above" and E may be "None of the Above".) So we have around 200 sections of the document, each with five opinions as to what category the section falls into.
The friend also classified each section herself, so we also have that sixth set of data. She believes that there are particular patterns to the sections, which has not been generally noticed by others in her field.
I've argued that we should not include her classification in any analysis, because it's her hypothesis -- the patterns she sees -- that we're looking to confirm, so we should only use the five other experts. Does this seem correct?
The test was originally given to seven experts, but two of them did not follow the instructions to pick a single category for each section. They gave answers like "A or B". Between the two of them, they chose multiple categories for 30% of the sections, so we chose not to use their answers. (My friend had also decided before giving the questionnaires to the subjects that any with more than 10% errors would not be used.) Does this seem like the correct choice?
Given the first two decisions, we decided that a section's category would be "strongly agreed to" if 4/5 or 5/5 of the experts put it into the same category. Does this seem reasonable, or should some kind of calculation be used to determine a level for strong agreement for each section?
Assuming that we decide we have enough sections with strongly-agreed-to categories to look for patterns, does anyone have pointers on the technique to use? For example, we might hypothesize that there is a pattern of sections of A followed by B followed by C (with perhaps E's or maybe D's thrown in). Is there a field or technique for this, or does it boil down to pattern matching and combinatorics?
I've created various contingency tables of experts/categories/sections/counts, done Cohen's Kappa, CA, MCA, etc, and it's all interesting, but perhaps the bottom line is that only about 1/3 of the sections have strong agreement (4/5 or 5/5 experts agree on the category), and I'm not sure that's enough coverage of the document to actually proceed to the next step and look for patterns. Any thoughts or suggestions about fields/techniques that might be helpful? (For example, I've briefly looked at IRT but it appears to not be useful without a gold standard and with nominal categorical data rather than ordinal.)