3
$\begingroup$

Basically I want to classify a rather rare status (about 2% of the 2000) with some predictors. I have used logistic regression, neural network, and Support Vector Machines to do it.

All the predictors in the logistic regression are statistically significant. And to avoid overfitting, I have self-implemented a 10-fold CV for all of the methods. For each iteration, I used the training dataset to fit the model and find out the fitted values. Than I use the ROCR package in R to find out the decision criterion to achieve 70% sensitivity for the training dataset. Then I use the model and the criterion to predict the status of the testing dataset and compute the testing sensitivity and positive predictive value. After 10 iterations I got 10 testing sensitivities and PPVs.

My finding is: the logistic regression did the best. The testing sensitivities were roughly 70% and PPVs were around 16%. But very surprisingly, the performance of SVM is very poor: mean testing sensitivity = 43%, PPV = 11%.

I am not very familiar with the theory behind SVM so I tried both kernlab and e1071 in R. I have also experimented with C-svc, nu-svc, C-bsvc, as well as tuning the svm using tune.svm in e1071 but the performance was similar.

So my question is: was I doing something wrong, or was I missing something when fitting a svm?

$\endgroup$
  • $\begingroup$ It sounds like you have a zero inflation problem. Given this, all of your specified models will be negatively effected. Have you tried a Zero Inflated Poisson (ZIP) model? In machine learning parlance this is an imbalanced class problem. $\endgroup$ – Jeffrey Evans Dec 4 '12 at 18:09
  • $\begingroup$ @JeffreyEvans Thanks for suggestions but my response variable is binary. Would it be appropriate to fit a ZIP as Poisson models are mainly for count data. $\endgroup$ – fred Dec 5 '12 at 1:42
  • $\begingroup$ You can use a zero-inflated negative binomial regression (ZIB). The "zeroinfl" function in the "pscl" package allows for a negative binomial distribution (dist="negbin"). $\endgroup$ – Jeffrey Evans Dec 5 '12 at 4:30
  • $\begingroup$ @JeffreyEvans as far as i know, zero-inflated NB model is still for count data but it just adjusts for the over-dispersion problem when compared to Poisson. $\endgroup$ – fred Dec 6 '12 at 13:38
  • $\begingroup$ Which kernel did you use when fitting the SVM? If you used a linear kernel it should be more or less the same as the results from the logistic regression. The SVM would instead then only calculate an "optimal" hyperplane, which should be more robust and therefor generalize better. $\endgroup$ – Dr. Mike Jan 23 '13 at 10:13
3
$\begingroup$

The SVM is designed to determine the optimal decision boundary for only one ratio of false-positive and false-negative misclassification costs, so it is not really a fair comparison to change the threshold to adjust the sensitivity. A better approach would be to tune the regularisation parameters (C or nu) for each class independently (some packages support this, some don't) by optimising a cross-validation estimate of your statistic of interest. Note that to get an unbiased performance estimate, you will need to perform a nested cross-validation.

Logistic regression doesn't suffer from this problem as the loss function is intended to minimise the error in estimating the posterior probability of class membership everywhere, rather than merely for p=0.5 (or some other value depending on the ratio of misclassification costs). I no longer use the SVM very much because for most applications, I do actually want the probabilities that logistic regression provides, however I use regularised logistic regression (which gives similar over-fitting avoidance you get with the SVM) and kernel logistic regression if I want a non-linear model (or Gaussian Process Classifiers for a Bayesian equivalent - although the difference in performance between GPC and KLR is generally quite small).

$\endgroup$
0
$\begingroup$

you can use the matlab codes for svm and compare your answers with that I think different packages give different answers and it sounds matlab is satisfactory. Here are codes for 10 fold cross validation in matlab:

load fisheriris %# load iris dataset groups = ismember(species,'setosa'); %# create a two-class problem

cvFolds = crossvalind('Kfold', groups, 10); %# get indices of 10-fold CV cp = classperf(groups); %# init performance tracker

for i = 1:10 %# for each fold
testIdx = (cvFolds == i); %# get indices of test instances
trainIdx = ~testIdx; %# get indices training instances

%# train an SVM model over training instances    
svmModel = svmtrain(meas(trainIdx,:), groups(trainIdx), ...       
             'Autoscale',true, 'Showplot',false, 'Method','QP', ...        
             'BoxConstraint',2e-1, 'Kernel_Function','rbf', 'RBF_Sigma',1);   

%# test using test instances  
pred = svmclassify(svmModel, meas(testIdx,:), 'Showplot',false);  

%# evaluate and update performance object
cp = classperf(cp, pred, testIdx);

end

%# get accuracy cp.CorrectRate

%# get confusion matrix %# columns:actual, rows:predicted, last-row: unclassified instances cp.CountingMatrix

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.