Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

Two tools completely agree on categorizing 85k sentences (my entire population). Now, I want to take a sample of sentences and give that to two humans. This is to check whether the tools agreed genuinely or both of them together miscategorized the 85k sentences. Now, I need to know how many sentences should I pick as a sample out of 85k sentences to check it statistically. Is there any formula or procedure to find out?

share|improve this question
The humans may disagree with eachother on their categorization, what would that mean? – Peter Ellis Jul 28 '12 at 7:56
The answer to your question may depend on the answers to other questions such as: (1) How many categories do you have? (You might want to check agreement for each category.) (2) Is there any reason to think that the categorizations will be easier or harder for some categories than others? (If you think there is no difference in difficulty across categories, perhaps you will waive a check on the agreement for each category.) (3) Are the baselines for each category the same? (For example, do a large majority of your cases fall in one category.) In short, can you tell us a bit more? – Joel W. Jul 29 '12 at 18:11

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.