I want to find patterns in the way that two models, Model A and Model B, perform on my dataset. What are some good ways to compare the individual predictions of the two models for $N$ classes? If any particular method is used for $N=2$, then that's interesting as well. I'm not interested in comparing the general accuracy, but in the individual predictions on my $ 100.000$ observations. A graphical comparison would be great, but other options would be nice as well.
For classification, the standard is the receiver operating characteristic (ROC) and/or precision vs recall plots.
If you're after a graphical comparison, then you can plot the confusion matrix / ROC or all/any three items in the image above.
First do this over all examples, for both models, plot the ROC.
Now, you said you want to find patterns. I can see two ways in which you can achieve this: 1) group examples somehow 2) logical operations on the predictions, repeat process. Figure out what it means.
For more insight you need to bin or accumulate the examples somehow. In sql terms, you group the data in batches. These should be meaningful. Clustering is one option where you can automatically (hopefully) extract meaning from looking at examples from each cluster.
For ordered data / time-series grouping can be on a basis of day, week, month or other scale. If there is any ordering in the examples, e.g. it comes as a time-series, then you can build one of these plots for each one of your subsets.
Other things to try, do these operations on the predictions of two models, plot the ROC again. Repeat for sub-groups.
This paper looks into improving prediction if more than one model is available: http://arxiv.org/pdf/1106.0219.pdf
This is done by finding patterns between models. Might be useful.
Main idea is to look intersections of different models. Where they agree, disagree and iterate over all four possible ways it is possible (similar to precision, recall etc). Given more than two models you can look intersections pairwise.