Visually compare predictions from multiple classifiers

Question

I have a classification problem where I have tested four different algorithms for binary classification. I have performed a 10-fold cross-validation on my training set and generated ROC curves and performance estimates (accuracy, AUC, etc).

I would now like to compare the classifiers at the level of the predictions to understand whether the decisions made by the different classifiers on individual observations are more or less the same.

Does anyone have suggestions/recommendations on how to do this visually?

A simple option would probably be to draw two 4-way Venn diagrams, one for the positive predictions and one for the negative, but I'm wondering if there are more elegant/clever ways to do it.

Thank you!

Answer 1

One possibility would be a "crosstable matrix", which tabulates the pairwise (in)congruences of your four classification algorithms' outputs, and displays all possibilities in a matrix. With some toy data in R:

require(gplots) # for textplot
nn <- 100
set.seed(1)
classification <- data.frame(A=runif(nn)<.3,B=runif(nn)<.3,C=runif(nn)<.3,D=runif(nn)<.3)

opar <- par(mfrow=rep(ncol(classification),2),mai=rep(0,4))
    for ( ii in 1:ncol(classification) ) {
        for ( jj in 1:ncol(classification) ) {
            if ( ii == jj ) {
                plot(c(0,1),c(0,1),type="n",xlab="",ylab="",bty="n",xaxt="n",yaxt="n")
                text(.5,.5,colnames(classification)[jj],cex=2)
            } else {
                textplot(table(classification[,c(ii,jj)]),xlab="",ylab="",bty="n",cex=0.9)
            }
        }
    }
par(opar)

Alternatively, you could visualize the entries of all these tables, e.g., using differently sized circles. (Make sure the area of these circles corresponds to the cell counts, not the radius.)