0
$\begingroup$

I'm trying to implement an automated system for spotting random selectors on some job. Simply, I'm trying to find out a scientific way to come up with a formula or rule to be implemented in this system.

The Scenario is: 1000 allegedly native German speakers are asked to look at a word and just decide if this word is in German. Given that 999 out of the 1000 people said that the word was in German, and only one said that it wasn't. The first thing that will come to one's mind that that person is not really a native German speaker, and he just selected the answer randomly.

Now, What would be the maximum ratio between the number of people who choose a specific answer to the rest who choose the opposite answer at which I can be "not 100% sure" but very very positive that those few people who choose the odd answer are selecting the answers randomly?

In other words, what's the maximum percentage of the odd answers compared to all answers, at which I can safely say that someone selects their answers randomly)? and what would be the formula/rule for this?

I strongly doubt it, but if statistics doesn't have an answer for this? is it even scientifically possible to come up with such solution?

$\endgroup$
  • 3
    $\begingroup$ It's not even clear that the question is well formulated. I can imagine English words, like "fardel" or "yclept," that would not be identified as English by a large majority of native English speakers--or possibly by all English speakers within a particular community. The lone person out of a million who does identify such a word as English might be the sole knowledgeable scholar in the group. How does that make her a "cheater"? $\endgroup$ – whuber Feb 23 '18 at 15:09
  • 1
    $\begingroup$ If by "cheater" you mean "someone with little or no knowledge of German", just given them a paragraph to translate from pretty much any popular novel in German. If by "cheater" you mean "not born in Germany", well, that's a very different thing, as millions of non-Germans are fluent in German. $\endgroup$ – jbowman Feb 23 '18 at 15:15
  • 1
    $\begingroup$ Any concern you could have 99 non-German speakers/bots/whatever and 1 real German speaker? $\endgroup$ – Björn Feb 23 '18 at 15:20
  • 5
    $\begingroup$ I feel compelled to upvote any comment containing the word yclept. $\endgroup$ – Stephan Kolassa Feb 23 '18 at 15:54
  • 1
    $\begingroup$ I don't understand this at all. Guessing randomly and cheating are not the same thing; I've guessed randomly (on specific questions about which I either had no clue or did not have time to solve) on many multiple choice tests! More fundamentally, I don't agree that there's a monotonic relationship between your input (percentage of "easy" questions answered incorrectly) and your output (classification as a "cheater"); indeed, there may be no relationship at all. We need more information about what you're trying to accomplish. $\endgroup$ – Josh Feb 23 '18 at 19:06
1
$\begingroup$

In its current form, this question is difficult to understand, and indeed fundamentally unanswerable because the certainty you seek cannot exist. I think what you are proposing is that you want to classify test takers into two categories:

1) Those answering test questions in good faith and using faculties commonly associated with human reasoning, and

2) Those answering test questions according to some algorithm; perhaps deliberately doing poorly to fake a disability.

The approach I would take is to establish a control group of good-faith test takers and, for each test taker, record the percentage $x_i$ of pre-identified and deliberately "easy" questions answered incorrectly by test taker $i$. Then, let $X-µ$ represent the (centered; i.e., zero-mean) vector of all the $x_i$ and compute its standard deviation $σ$. Then the z-score corresponding to a suspected cheater's percentage $x_{cheat}$ would be given by $(x_{cheat}-µ)/σ$. This is the distance, in units of $σ$, that $x_{cheat}$ lies from the mean of $X$. Large positive values would tend to suggest that there may be some algorithmic test-taking going on.

Caveat: If your "easy" questions are too easy, then the distribution of $X$ will wind up saturating at zero and become highly asymmetric; i.e., not at all Gaussian. If that's the case, then you'll need to be very careful about drawing conclusions according to this method.

Edit: Ideally, you'd have both categories of test takers in your control group (with their identities known). That would allow you to create a classification model from these data. However, I took your question to be about associating scores with levels of suspicion of cheating, as there was no explicit mention of modeling.

$\endgroup$
  • $\begingroup$ Thanks Josh. As I mentioned in my comment to your comment above, we van just totally drop the "cheater" word and choose "random selector". Another clarification is that I'm not seeking certainty in my decision, I just want to have an idea about the highest possible ratio between ones with odd answers, and those who gave the same answer of majority of the test takers, at which I can say there is really really a huge possibility that those with the odd answers were just randomly picking answers. One last thing, would you kindly simplify your answer as I'm not a statistician. $\endgroup$ – Folan Feb 23 '18 at 21:55
  • $\begingroup$ You can't deal in these sorts of matters without using some statistics; sorry (I'm not a statistician either, BTW). "Really really a huge possibility" needs to be quantified; perhaps some sort of z-score cutoff would be appropriate. There's no easy way out. I think this is about as simple as it gets. If something in particular is confusing, let me know and I'll try to explain it. $\endgroup$ – Josh Feb 23 '18 at 22:11
1
$\begingroup$

If you ask multiple words ranging from easy to difficult, then the easy ones can be your filter. You can eliminate the people that get the monkey score not only for the difficult ones, but also the easy ones.

Depending one your specific desires and research question there are different ways to have this implemented in your questionnaire and possibly add other diagnostic features (e.g. reaction time, use blocks of questions, ...).

$\endgroup$
  • $\begingroup$ Thanks @Martijn "You can eliminate the people that get the monkey score". Well, this is exactly what I need to decide. if all question are simply true or false, then how can I define "monkey score"? $\endgroup$ – Folan Feb 24 '18 at 16:48
  • $\begingroup$ a monkey would score correct on half of your questions. The deviation can be found by using the binomial distribution. Some monkeys might still get all questions correct or at least surpass a level such that they are indistinguishable from people that do not fill in the questions at random. So your problem may actually be not so much 'whether you should consider this person with the wrong answer as a non-German speaking panelist who is faking it', and instead 'how many there may be that guessed correctly and were not detected'. $\endgroup$ – Martijn Weterings Feb 26 '18 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.