I'm trying to implement an automated system for spotting random selectors on some job. Simply, I'm trying to find out a scientific way to come up with a formula or rule to be implemented in this system.
The Scenario is: 1000 allegedly native German speakers are asked to look at a word and just decide if this word is in German. Given that 999 out of the 1000 people said that the word was in German, and only one said that it wasn't. The first thing that will come to one's mind that that person is not really a native German speaker, and he just selected the answer randomly.
Now, What would be the maximum ratio between the number of people who choose a specific answer to the rest who choose the opposite answer at which I can be "not 100% sure" but very very positive that those few people who choose the odd answer are selecting the answers randomly?
In other words, what's the maximum percentage of the odd answers compared to all answers, at which I can safely say that someone selects their answers randomly)? and what would be the formula/rule for this?
I strongly doubt it, but if statistics doesn't have an answer for this? is it even scientifically possible to come up with such solution?