Comparing classifier performance

Question

I used a SVM to perform classifications on the same sample and I get the following results:

Good classifications:

        baseline : 78%
transformation 1 : 79%
transformation 2 : 63%

I do not know which test to use to tell if the performance difference is significant (ie. is the transformation 1 really better than transformation 2?). Which is the adequate test?

Note: transformations are transformations performed on datas (i.e. : different imputation method for missing values).

Answer 1

As usual, I would propose to cross-validate both models (not only use a single test) and use the data from individual folds to get the spread required for making an equality test.

Of course this method requires to repeat the imputation in each fold.

Answer 2

I don't know of ways to find SEs for SVM (they're not my thing), but if you want to compare the imputation methods, you can at least do each imputation 1000 times and find an estimate for the SE of the correct classification percentage from the variance over your repeated imputations.

To be correct, you'd need to add a variance component from the method (SVM) itself, but if the difference is already smaller than (the desired multiple) of your SE, you know the difference is not significant.

Little and Rubin provide this approach in much more detail.