This question regards the best way to compare ordered binary data. My situation is as follows:

I'm interested in evaluating how well a model conforms to human performance data on a set of validation problems. For a test set of 30 problems, I have a record of whether my model produced a correct response (denoted as a "1") or no response at all** (a "0") within a time limit. Because the model operates probabilistically, I have data for multiple runs of the model on the same test set (organized in a binary matrix of run # vs. problem #). I'd like to compare this accuracy data against the accuracy data produced by a group of human participants working on the same validation set. My question is: what test(s) can I use to measure the degree of coherence between the two datasets that takes into account the different questions in the validation set (i.e., that recognizes that data in the first column of the model accuracy matrix [corresponding to performance on the first question in the validation set] should only be compared against data in the first column of the human accuracy matrix, etc.)?

Apologies if this has been answered elsewhere!

** Because the model is structured such that it never generates a response unless it is the correct one (i.e. it simply keeps running until it either (1) finds the correct answer or (2) runs out of time, whichever comes first), the response output doesn't include false positives or false negatives (so far as I understand those terms).

  • $\begingroup$ This question seems astounding to me. I find it hard to believe you could have a model with zero false positives. What is the data being validated? $\endgroup$ – naught101 Aug 17 '13 at 1:19
  • $\begingroup$ @naught101 Perhaps my understanding of false positive is false (har har). What the model does NOT produce is an incorrect solution to a question that it labels as correct (e.g., erroneously producing an output of 5 to the question what is 2+2?). The model only produces an output if the output can be formally verified. This is similar to a mathematician generating a solution to a problem and then checking their work to see whether their solution is mathematically sound. If the model cannot generate a solution in this way within the time limit, it simply doesn't generate any output. $\endgroup$ – StuartBernis Aug 17 '13 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.