Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite
1

A friend of mine took a document and broke it up into parts, then asked 5 subject matter experts to classify each part into nominal category A, B, C, D, or E. (I'm not sure yet, but D may be "All of the Above" and E may be "None of the Above".) So we have around 200 sections of the document, each with five opinions as to what category the section falls into.

The friend also classified each section herself, so we also have that sixth set of data. She believes that there are particular patterns to the sections, which has not been generally noticed by others in her field.

Questions:

  1. I've argued that we should not include her classification in any analysis, because it's her hypothesis -- the patterns she sees -- that we're looking to confirm, so we should only use the five other experts. Does this seem correct?

  2. The test was originally given to seven experts, but two of them did not follow the instructions to pick a single category for each section. They gave answers like "A or B". Between the two of them, they chose multiple categories for 30% of the sections, so we chose not to use their answers. (My friend had also decided before giving the questionnaires to the subjects that any with more than 10% errors would not be used.) Does this seem like the correct choice?

  3. Given the first two decisions, we decided that a section's category would be "strongly agreed to" if 4/5 or 5/5 of the experts put it into the same category. Does this seem reasonable, or should some kind of calculation be used to determine a level for strong agreement for each section?

  4. Assuming that we decide we have enough sections with strongly-agreed-to categories to look for patterns, does anyone have pointers on the technique to use? For example, we might hypothesize that there is a pattern of sections of A followed by B followed by C (with perhaps E's or maybe D's thrown in). Is there a field or technique for this, or does it boil down to pattern matching and combinatorics?

  5. I've created various contingency tables of experts/categories/sections/counts, done Cohen's Kappa, CA, MCA, etc, and it's all interesting, but perhaps the bottom line is that only about 1/3 of the sections have strong agreement (4/5 or 5/5 experts agree on the category), and I'm not sure that's enough coverage of the document to actually proceed to the next step and look for patterns. Any thoughts or suggestions about fields/techniques that might be helpful? (For example, I've briefly looked at IRT but it appears to not be useful without a gold standard and with nominal categorical data rather than ordinal.)

share|improve this question
1  
"Pattern" is such a general concept that I'm certain you could find some kind of "patterns" even if five untrained monkeys did the classification, provided you look hard enough. If you have a more specific sense of what kinds of patterns you are looking for, that could help focus the investigation and it might even suggest good data analysis techniques. – whuber♦ Dec 31 '12 at 19:57
@whuber: I'm with you on that. All I can say is that the friend believes that this document's sections tend to fall into a logical pattern of A then B then (and implies) C. Yeah, the "implies" part is a whole other mess. It's not just "clustering" A, B, and C. Strikes me as something pretty ill-formed, but we're currently at the stage of looking at the ways the experts categorized things, seeing if we either have enough agreement to proceed to the tricky part, or if we can figure out what we might change and re-test (like the categories) to get better agreement (hence CA/MCA, etc). – Wayne Dec 31 '12 at 20:32
@whuber: any thoughts on questions 1-3? – Wayne Dec 31 '12 at 20:33
I think it would be possible to accommodate the uncertain experts and you would want to do this because it increases the amount of data substantially. However, how you accommodate them depends on what form the analysis takes. Your description of "A then B then C" sounds like a two-stage Markov model might be worth exploring to see whether the transition probabilities A-->B and (A,B)-->C are relatively large. – whuber♦ Dec 31 '12 at 21:32
@whuber: please turn your comment into an answer, so I can choose it. I like the Markov model approach for the second step, and I think you're saying that if we use a Markov model the "A or B" answers naturally become part of the transition probabilities. Which would be a win-win in terms of the data we have and the approach. – Wayne Dec 31 '12 at 22:25

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.