Basically I want to classify a rather rare status (about 2% of the 2000) with some predictors. I have used logistic regression, neural network, and Support Vector Machines to do it.
All the predictors in the logistic regression are statistically significant. And to avoid overfitting, I have self-implemented a 10-fold CV for all of the methods. For each iteration, I used the training dataset to fit the model and find out the fitted values. Than I use the
ROCR package in R to find out the decision criterion to achieve 70% sensitivity for the training dataset. Then I use the model and the criterion to predict the status of the testing dataset and compute the testing sensitivity and positive predictive value. After 10 iterations I got 10 testing sensitivities and PPVs.
My finding is: the logistic regression did the best. The testing sensitivities were roughly 70% and PPVs were around 16%. But very surprisingly, the performance of SVM is very poor: mean testing sensitivity = 43%, PPV = 11%.
I am not very familiar with the theory behind SVM so I tried both
e1071 in R. I have also experimented with
C-bsvc, as well as tuning the svm using
e1071 but the performance was similar.
So my question is: was I doing something wrong, or was I missing something when fitting a svm?