I have developed a web application in which I show a picture and two captions. One of those captions is the correct and the other is randomly selected from a corpus of picture captions.
Next, I present the following question: "Which caption best describes the picture?" and the user casts a vote on the caption that he chooses.
Since I store the user votes, I know how many votes each picture received and also, the fraction of votes that were on the correct caption.
Since the 'correct' captions of some pictures are misleading, it is expected that, in these cases, the correct vote fraction is near %50 (that is, it is impossible to identify the correct caption). However, some pictures will have very descriptive captions, and the correct vote fraction is expected to be somewhat higher.
Is it possible, given a confidence value (e.g. 95%), to identify pictures whose correct caption is impossible for the users to identify (correct vote fraction = %50)?