I am trying to compare the classification performance of different classifiers. So far, I am using SVM, Random forest, Adaboost.M1, and Naive Bayes. 70% of the data is used for training (and then plotting the ROC curve), while 30% is used for testing (a ROC curve again). Being fairly new to ML and more specifically to ROC curves, I have the following questions:
Can we compare classifiers using the ROC curve? If so, the discussion will be on AUC? In this case, should I discuss accuracy, f-measure...
Should I evaluate using a ROC curve based on the training data or on the test data? Which is more meaningful? And why?
In my simulation, my testing ROC curve is different from training (specifically for SVM classifier). SVM's training ROC curve is very good as compared to testing data curve (see below). Is this correct? If so, how should I analyze/present it? Pointers would be really helpful. The top one is based on the training data, where SVM's curve is very good. The bottom one is based on testing data, where SVM is not so good, and is being outperformed by random forests. So should I say that random forests is doing a good job?