This question regards the best way to compare ordered binary data. My situation is as follows:
I'm interested in evaluating how well a model conforms to human performance data on a set of validation problems. For a test set of 30 problems, I have a record of whether my model produced a correct response (denoted as a "1") or no response at all** (a "0") within a time limit. Because the model operates probabilistically, I have data for multiple runs of the model on the same test set (organized in a binary matrix of run # vs. problem #). I'd like to compare this accuracy data against the accuracy data produced by a group of human participants working on the same validation set. My question is: what test(s) can I use to measure the degree of coherence between the two datasets that takes into account the different questions in the validation set (i.e., that recognizes that data in the first column of the model accuracy matrix [corresponding to performance on the first question in the validation set] should only be compared against data in the first column of the human accuracy matrix, etc.)?
Apologies if this has been answered elsewhere!
** Because the model is structured such that it never generates a response unless it is the correct one (i.e. it simply keeps running until it either (1) finds the correct answer or (2) runs out of time, whichever comes first), the response output doesn't include false positives or false negatives (so far as I understand those terms).