5
$\begingroup$

I have a classification problem where I have tested four different algorithms for binary classification. I have performed a 10-fold cross-validation on my training set and generated ROC curves and performance estimates (accuracy, AUC, etc).

I would now like to compare the classifiers at the level of the predictions to understand whether the decisions made by the different classifiers on individual observations are more or less the same.

Does anyone have suggestions/recommendations on how to do this visually?

A simple option would probably be to draw two 4-way Venn diagrams, one for the positive predictions and one for the negative, but I'm wondering if there are more elegant/clever ways to do it.

Thank you!

$\endgroup$

1 Answer 1

2
$\begingroup$

One possibility would be a "crosstable matrix", which tabulates the pairwise (in)congruences of your four classification algorithms' outputs, and displays all possibilities in a matrix. With some toy data in R:

require(gplots) # for textplot
nn <- 100
set.seed(1)
classification <- data.frame(A=runif(nn)<.3,B=runif(nn)<.3,C=runif(nn)<.3,D=runif(nn)<.3)

opar <- par(mfrow=rep(ncol(classification),2),mai=rep(0,4))
    for ( ii in 1:ncol(classification) ) {
        for ( jj in 1:ncol(classification) ) {
            if ( ii == jj ) {
                plot(c(0,1),c(0,1),type="n",xlab="",ylab="",bty="n",xaxt="n",yaxt="n")
                text(.5,.5,colnames(classification)[jj],cex=2)
            } else {
                textplot(table(classification[,c(ii,jj)]),xlab="",ylab="",bty="n",cex=0.9)
            }
        }
    }
par(opar)

crosstable matrix

Alternatively, you could visualize the entries of all these tables, e.g., using differently sized circles. (Make sure the area of these circles corresponds to the cell counts, not the radius.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.