Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite

I have developed a web application in which I show a picture and two captions. One of those captions is the correct and the other is randomly selected from a corpus of picture captions.

Next, I present the following question: "Which caption best describes the picture?" and the user casts a vote on the caption that he chooses.

Since I store the user votes, I know how many votes each picture received and also, the fraction of votes that were on the correct caption.

Since the 'correct' captions of some pictures are misleading, it is expected that, in these cases, the correct vote fraction is near %50 (that is, it is impossible to identify the correct caption). However, some pictures will have very descriptive captions, and the correct vote fraction is expected to be somewhat higher.

Is it possible, given a confidence value (e.g. 95%), to identify pictures whose correct caption is impossible for the users to identify (correct vote fraction = %50)?

share|improve this question
Won't there tend to be some bias in the case where there isn't really a correct answer. For example, you give a choice of Cindy Crawford and Nell Carter to describe a picture of the Sun.. Most people might just pick Cindy Crawford cause they like her. Just a thought. – Adam Oct 11 '11 at 2:40

1 Answer

up vote 1 down vote

You could construct a confidence interval for the frequency of correct guesses, or could test the hypothesis that the frequency is 1/2. There's a quite complete discussion of CI/tests for binomial proportions at wikipedia.

In R, I'd use the function binom.test.

share|improve this answer
Just to add on to @Karl's answer, I would pair that with a Bonferroni/false discovery rate correction to account for the multiple comparisons. – Charlie Oct 11 '11 at 2:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.