10
$\begingroup$

I am using SVM to predict diabetes. I am using the BRFSS data set for this purpose. The data set has the dimensions of $432607 \times 136$ and is skewed. The percentage of Ys in the target variable is $11\%$ while the Ns constitute the remaining $89\%$.

I am using only 15 out of 136 independent variables from the data set. One of the reasons for reducing the data set was to have more training samples when rows containing NAs are omitted.

These 15 variables were selected after running statistical methods such as random trees, logistic regression and finding out which variables are significant from the resulting models. For example, after running logistic regression we used p-value to order the most significant variables.

Is my method of doing variable selection correct? Any suggestions to is greatly welcome.

The following is my R implementation.

library(e1071) # Support Vector Machines

#--------------------------------------------------------------------
# read brfss file (huge 135 MB file)
#--------------------------------------------------------------------
y <- read.csv("http://www.hofroe.net/stat579/brfss%2009/brfss-2009-clean.csv")
indicator <- c("DIABETE2", "GENHLTH", "PERSDOC2", "SEX", "FLUSHOT3", "PNEUVAC3", 
    "X_RFHYPE5", "X_RFCHOL", "RACE2", "X_SMOKER3", "X_AGE_G", "X_BMI4CAT", 
    "X_INCOMG", "X_RFDRHV3", "X_RFDRHV3", "X_STATE");
target <- "DIABETE2";
diabetes <- y[, indicator];

#--------------------------------------------------------------------
# recode DIABETE2
#--------------------------------------------------------------------
x <- diabetes$DIABETE2;
x[x > 1]  <- 'N';
x[x != 'N']  <- 'Y';
diabetes$DIABETE2 <- x; 
rm(x);

#--------------------------------------------------------------------
# remove NA
#--------------------------------------------------------------------
x <- na.omit(diabetes);
diabetes <- x;
rm(x);

#--------------------------------------------------------------------
# reproducible research 
#--------------------------------------------------------------------
set.seed(1612);
nsamples <- 1000; 
sample.diabetes <- diabetes[sample(nrow(diabetes), nsamples), ]; 

#--------------------------------------------------------------------
# split the dataset into training and test
#--------------------------------------------------------------------
ratio <- 0.7;
train.samples <- ratio*nsamples;
train.rows <- c(sample(nrow(sample.diabetes), trunc(train.samples)));

train.set  <- sample.diabetes[train.rows, ];
test.set   <- sample.diabetes[-train.rows, ];

train.result <- train.set[ , which(names(train.set) == target)];
test.result  <- test.set[ , which(names(test.set) == target)];

#--------------------------------------------------------------------
# SVM 
#--------------------------------------------------------------------
formula <- as.formula(factor(DIABETE2) ~ . );
svm.tune <- tune.svm(formula, data = train.set, 
    gamma = 10^(-3:0), cost = 10^(-1:1));
svm.model <- svm(formula, data = train.set, 
    kernel = "linear", 
    gamma = svm.tune$best.parameters$gamma, 
    cost  = svm.tune$best.parameters$cost);

#--------------------------------------------------------------------
# Confusion matrix
#--------------------------------------------------------------------
train.pred <- predict(svm.model, train.set);
test.pred  <- predict(svm.model, test.set);
svm.table <- table(pred = test.pred, true = test.result);
print(svm.table);

I ran with $1000$ (training = $700$ and test = $300$) samples since it is faster in my laptop. The confusion matrix for the test data ($300$ samples) I get is quite bad.

    true
pred   N   Y
   N 262  38
   Y   0   0

I need to improve my prediction for the Y class. In fact, I need to be as accurate as possible with Y even if I perform poorly with N. Any suggestions to improve the accuracy of classification would be greatly appreciated.

$\endgroup$
  • $\begingroup$ I guess your SVM doesn't work at all, but don't know why ! also may be its better to normalized your data ... $\endgroup$ – user4581 Aug 13 '11 at 15:44
  • $\begingroup$ Yes it basically predicts Y for any input. This means it will be correct $~\approx 90\%$ of the time. $\endgroup$ – Anand Aug 13 '11 at 17:03
  • $\begingroup$ Normalize the data is the best thing to start. Begin by that. You can also try to search also non-linear kernel may should show a better result. (It depends of your frontier prevision, may be normalizing should be enough) $\endgroup$ – 404Dreamer_ML Aug 22 '11 at 17:44
  • $\begingroup$ You may also try kernlab instead of e1071 -- it does normalization automatically and has some heuristics making easier to bootstrap the first model. $\endgroup$ – user88 Nov 11 '11 at 16:18
9
$\begingroup$

I have 4 suggestions:

  1. How are you choosing the variables to include in your model? Maybe you are missing some the key indicators from the larger dataset.
  2. Almost all of the indicators you are using (such as sex, smoker, etc.) should be treated as factors. Treating these variables as numeric is wrong, and is probably contributing to the error in your model.
  3. Why are you using an SVM? Did you try any simpler methods, such as linear discriminant analysis or even linear regression? Maybe a simple approach on a larger dataset will yield a better result.
  4. Try the caret package. It will help you cross-validate model accuracy, it is parallelized which will let you work faster, and it makes it easy to explore different types of models.

Here is some example code for caret:

library(caret)

#Parallize
library(doSMP)
w <- startWorkers()
registerDoSMP(w)

#Build model
X <- train.set[,-1]
Y <- factor(train.set[,1],levels=c('N','Y'))
model <- train(X,Y,method='lda')

#Evaluate model on test set
print(model)
predY <- predict(model,test.set[,-1])
confusionMatrix(predY,test.set[,1])
stopWorkers(w)

This LDA model beats your SVM, and I didn't even fix your factors. I'm sure if you recode Sex, Smoker, etc. as factors, you will get better results.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I get the following error task 1 failed - "could not find function "predictionFunction"". I know this is not a forum but if you have any comments please let me know. $\endgroup$ – Anand Nov 23 '11 at 1:51
  • 1
    $\begingroup$ @Anand: Open a new R session as admin (or run sudo R on mac/linux). Run update.packages. When that is finished, close R and re-open a normal (non-admin) session. Run your code except for the "SVM" and "Confusion matrix" sections. Then run my code. If you still get an error, please post the line that returned an error, along with the exact error. $\endgroup$ – Zach Nov 23 '11 at 14:21
  • 1
    $\begingroup$ @Anand: Also, make sure you are running the latest version of R (2.14), and using the latest version of caret. You can update caret by running install.packages('caret') again. $\endgroup$ – Zach Nov 23 '11 at 14:23
  • $\begingroup$ @Anand: Great! You can drop in different methods for the train function, such as nb (naive bayes), glm (logistic regression), svmLinear and svmRadial. The svm's will take a long time to fit. $\endgroup$ – Zach Nov 23 '11 at 19:02
3
$\begingroup$

If you are using a linear kernel, then it is possible that feature selection is a bad idea, and that regularisation can prevent over-fitting more effectively than feature selection can. Note that the performance bounds that the SVM approximately implements are independent of the dimension of the feature space, which was one of the selling points of the SVM.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I've had this problem recently and found a couple of things that help. First, try out a Naive Bayes model (package klaR) which sometimes gives you better results when the minority class in a classification problem is tiny. Also, if you do choose to stick with an SVM you might want to try oversampling the minority class. Essentially you'll want to include more examples of the minority class or synthetically create cases for the minority class

This paper :http://www.it.iitb.ac.in/~kamlesh/Page/Reports/highlySkewed.pdf

Has some discussion and examples of these techniques implemented in Weka, but implimenting them yourself in R is possible too.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the helpful comments. Let me try out your suggestions. $\endgroup$ – Anand Nov 13 '11 at 4:51
1
$\begingroup$

In addition to what has already been mentioned, you are fixing your best model to use a linear kernel. You should predict using the best model that was tuned, including the same kernel that was used/found in your tuning stage (which I assume is RBF since your are tuning gamma).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.