9
$\begingroup$

Suppose an invigilator suspects one student of copying answers off another student's paper during a multiple-choice exam. She later checks their answers and finds some similarities—but on the other hand, there are bound to be similarities given the nature of the exam. How should she go about determining whether her suspicions were founded?

In other words, she will surely have to compare the exams to those of other students (who, let us assume, were not cheating). But if the class size is very large, is it reasonable to take a random sampling for comparison? How many would she then take? If there were many questions on the exam, would it also be reasonable to take a sampling of the questions for comparison? Does it make a significant difference whether each question had 2 possible answers (true/false) or, say, 4?

I don't have any specific numbers because I'm wondering about how this would work in general. I have a background in mathematics but little training in statistics. How would you describe this analysis in statistical terms?

Thank you.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ I have a feeling you have to make the assumption here that neither cheater nor cheatee had a majority correct answers. For example, if both of them got correct answers all around, you cant prove anything. But say both of them got the same wrong answers all around, there is probably very high likelihood of cheating. I think you will have to concentrate on answers that were wrong to do this measurement. $\endgroup$
    – Spacey
    Commented Aug 31, 2012 at 19:42
  • $\begingroup$ I would think that you might want to be selective and pick questions that are most likely to be copied. Those would probably be the ones that appear to be the most difficult. But there is also the possibility that the person cheating is just picking questions which covered topics he or she did not study and those would be difficult to discern. But having the same answers on easy questions really wouldn't tell you anything as both parties would know the correct answer. $\endgroup$
    – Michael R. Chernick
    Commented Aug 31, 2012 at 19:47
  • 2
    $\begingroup$ Not surprisingly, a lot of people have looked at cheating detection in the past, including Steven Levitt, author of Freakonomics. If you want to be able to tell whether someone cheated from the answers alone, don't give multiple choice tests, and proctor the exams yourself. You might be able to reject the hypothesis that the students' work was unrelated, but you will have a terrible time proving that they didn't simply study together. Do you have a seating chart and did you verify the students' IDs, that they were sitting according to the seating chart? Can you retest the students? $\endgroup$
    – Douglas Zare
    Commented Aug 31, 2012 at 19:56
  • 1
    $\begingroup$ Sampling the questions looks like a terrible idea since you can easily analyze all questions, and you will miss great indicators of copying such as a string of answers which are offset by 1 from the correct answer. E.g., The correct answers are 30) A 31) B 32) C 33) D 34)E and one student has 30) A 31) B 32) C 33) D 34) B, and another has 30) B 31) C 32) D 33) B. If these answers are very unpopular incorrect answers, then they fit the model that the second student was copying the first, and made an omission error. It is hard, though possible, to explain these answers without copying. $\endgroup$
    – Douglas Zare
    Commented Aug 31, 2012 at 20:02
  • 1
    $\begingroup$ With current software, it is relatively easy and efficient to create a set of exams with the same questions, but both the order of the questions and the order of the answers are permuted. Generally you only need at most is 4 versions. $\endgroup$
    – R. Schumacher
    Commented Aug 31, 2012 at 21:06

1 Answer 1

Reset to default
8
$\begingroup$

Here's a surprisingly vast array of the answer copying indexes, with little discussion of their merits though: http://www.bjournal.co.uk/paper/BJASS_01_01_06.pdf.

There's a field of (educational) psychology called item response theory (IRT) that provides the statistical background for questions like these. If you an American, and took an SAT, ACT or GRE, you dealt with a test developed with IRT in mind. The basic postulate of IRT is that each student $i$ is characterized by their ability $a_i$; each question is characterized by its difficulty $b_j$; and the probability to answer a question correctly is $$ \pi(a_i,b_j;c) = {\rm Prob}[\mbox{student $i$ answers question $j$ correctly}] = \Phi( c(a_i-b_j) ) $$ where $\Phi(z)$ is the cdf of the standard normal, and $c$ is an additional sensitivity/discrimination parameter (sometimes, it is made question-specific, $c_j$, if there's enough information, i.e., enough test takers, to identify the differences). A hidden assumption here that given the students ability $i$, answers to different questions are independent. This assumption is violated if you have a battery of questions about say the same paragraph of text, but let's abstract from it for a minute.

For "Yes/No" questions, this may be the end of the story. For more than two category questions, we can make an additional assumption that all wrong choices are equally likely; for a question $j$ with $k_j$ choices, probability of each wrong choice is $\pi'(a_i,b_j;c) = [1-\pi(a_i,b_j;c)]/(k_j-1)$.

For students of abilities $a_i$ and $a_k$, the probability that they match on their answers for a question with difficulty $b_j$ is $$ \psi(a_i,a_k;b_j,c) = \pi(a_i,b_j;c)\pi(a_k,b_j;c) + (k-1)\pi'(a_i,b_j;c)\pi'(a_k,b_j;c) $$ If you like, you can break this into probability of matching on the correct answer, $\psi_c(a_i,a_k;b_j,c) = \pi(a_i,b_j;c)\pi(a_k,b_j;c)$, and the probability of matching on an incorrect answer, $\psi_i(a_i,a_k;b_j,c) = (k-1)\pi'(a_i,b_j;c)\pi'(a_k,b_j;c)$, although from the conceptual framework of IRT, this distinction is hardly material.

Now, you can compute the probability of matching, but it will probably be combinatorially minuscule. A better measure may be the ratio of the information in the pairwise pattern of responses, $$ I(i,k) = \sum_j 1\{ \mbox{match}_j \} \ln \psi(a_i,a_k;b_j,c) + 1\{ \mbox{non-match}_j \} \ln [1- \psi(a_i,a_k;b_j,c) ] $$ and relate it to the entropy $$ E(i,k) = {\rm E}[ I(i,k) ] = \sum_j \psi(a_i,a_k;b_j,c) \ln \psi(a_i,a_k;b_j,c) + (1- \psi(a_i,a_k;b_j,c) ) \ln [1- \psi(a_i,a_k;b_j,c) ] $$ You can do this for all pairs of students, plot them or rank them, and investigate the greatest ratios of information to entropy.

The parameters of the test $\{c,b_j, j=1, 2, \ldots\}$ and student abilities $\{a_i\}$ won't fall out of blue sky, but they are easily estimable in modern software such as R with lme4 or similar packages:

    irt <- glmer( answer ~ 1 + (1|student) + (1|question), family = binomial)

or something very close to this.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.