What is the highest possible ratio to be almost certain that someone is randomly selecting answers on a test?

Question

I'm trying to implement an automated system for spotting random selectors on some job. Simply, I'm trying to find out a scientific way to come up with a formula or rule to be implemented in this system.

The Scenario is: 1000 allegedly native German speakers are asked to look at a word and just decide if this word is in German. Given that 999 out of the 1000 people said that the word was in German, and only one said that it wasn't. The first thing that will come to one's mind that that person is not really a native German speaker, and he just selected the answer randomly.

Now, What would be the maximum ratio between the number of people who choose a specific answer to the rest who choose the opposite answer at which I can be "not 100% sure" but very very positive that those few people who choose the odd answer are selecting the answers randomly?

In other words, what's the maximum percentage of the odd answers compared to all answers, at which I can safely say that someone selects their answers randomly)? and what would be the formula/rule for this?

I strongly doubt it, but if statistics doesn't have an answer for this? is it even scientifically possible to come up with such solution?

Answer 1

In its current form, this question is difficult to understand, and indeed fundamentally unanswerable because the certainty you seek cannot exist. I think what you are proposing is that you want to classify test takers into two categories:

1) Those answering test questions in good faith and using faculties commonly associated with human reasoning, and

2) Those answering test questions according to some algorithm; perhaps deliberately doing poorly to fake a disability.

The approach I would take is to establish a control group of good-faith test takers and, for each test taker, record the percentage $x_i$ of pre-identified and deliberately "easy" questions answered incorrectly by test taker $i$. Then, let $X-µ$ represent the (centered; i.e., zero-mean) vector of all the $x_i$ and compute its standard deviation $σ$. Then the z-score corresponding to a suspected cheater's percentage $x_{cheat}$ would be given by $(x_{cheat}-µ)/σ$. This is the distance, in units of $σ$, that $x_{cheat}$ lies from the mean of $X$. Large positive values would tend to suggest that there may be some algorithmic test-taking going on.

Caveat: If your "easy" questions are too easy, then the distribution of $X$ will wind up saturating at zero and become highly asymmetric; i.e., not at all Gaussian. If that's the case, then you'll need to be very careful about drawing conclusions according to this method.

Edit: Ideally, you'd have both categories of test takers in your control group (with their identities known). That would allow you to create a classification model from these data. However, I took your question to be about associating scores with levels of suspicion of cheating, as there was no explicit mention of modeling.

Answer 2

If you ask multiple words ranging from easy to difficult, then the easy ones can be your filter. You can eliminate the people that get the monkey score not only for the difficult ones, but also the easy ones.

Depending one your specific desires and research question there are different ways to have this implemented in your questionnaire and possibly add other diagnostic features (e.g. reaction time, use blocks of questions, ...).