# Using hypothesis-testing to identify misleading picture captions

I have developed a web application in which I show a picture and two captions. One of those captions is the correct and the other is randomly selected from a corpus of picture captions.

Next, I present the following question: "Which caption best describes the picture?" and the user casts a vote on the caption that he chooses.

Since I store the user votes, I know how many votes each picture received and also, the fraction of votes that were on the correct caption.

Since the 'correct' captions of some pictures are misleading, it is expected that, in these cases, the correct vote fraction is near %50 (that is, it is impossible to identify the correct caption). However, some pictures will have very descriptive captions, and the correct vote fraction is expected to be somewhat higher.

Is it possible, given a confidence value (e.g. 95%), to identify pictures whose correct caption is impossible for the users to identify (correct vote fraction = %50)?

-
Won't there tend to be some bias in the case where there isn't really a correct answer. For example, you give a choice of Cindy Crawford and Nell Carter to describe a picture of the Sun.. Most people might just pick Cindy Crawford cause they like her. Just a thought. –  Adam Oct 11 '11 at 2:40
In R, I'd use the function binom.test.