# Comparing classifier performance

I used a SVM to perform classifications on the same sample and I get the following results:

Good classifications:

        baseline : 78%
transformation 1 : 79%
transformation 2 : 63%


I do not know which test to use to tell if the performance difference is significant (ie. is the transformation 1 really better than transformation 2?). Which is the adequate test?

Note: transformations are transformations performed on datas (i.e. : different imputation method for missing values).

-
Are these figures classification accuracy on an independent test set? How big is the test set? –  DavidDLewis Aug 31 '11 at 12:23

As usual, I would propose to cross-validate both models (not only use a single test) and use the data from individual folds to get the spread required for making an equality test.

Of course this method requires to repeat the imputation in each fold.

-