6
$\begingroup$

For my classification problem, I am trying to classify an object as Good or Bad. I have been able to create a good first classification step that separates the data into 2 groups using SVM.

After tuning the parameters for the SVM using a training/holdout set (75% training, 25% holdout), I obtained the following results from the holdout set: Group 1 (model classified as Bad) consisted of 99% Bad objects, and Group 2 (model classified as Good) consisted of about 45% Good objects and 55% Bad objects. I verified the performance of the model using k-fold CV (k=5) and found the model to be stable and perform relatively consistently in terms of misclassification rates.

Now, I want to pass these objects through another round of classification by training another model (may or may not be SVM) on my group 2 of maybe good/maybe bad objects to try and correctly classify this second group now that I have gotten rid of the obviously bad objects.

I had a couple of thoughts, but am unsure of how to proceed.

(1) My first idea was to use the data from the classified objects from the Holdout set to train another model. I was able to train another classification model from the results of the holdout set. The problem is I am using less than 25% of the original data, and I am worried of overfitting on a very small subset of my data.

(2) My second idea was to gather the results of the 5-fold CV to create another dataset. My reasoning is that since the data is partitioned into 5 parts, and each part is classified into two groups from a model trained by the other 4 parts, I thought that I could aggregate the predicted results of the 5 parts to obtain a classified version of my original dataset and continue from there.

The only problem is, I have a sinking feeling that both methods are no good. Could CV shed some light on some possible next steps?

EDIT

Sorry, my question was badly worded. Let me try to clarify what I am trying to do. It can be thought of like a tree...

  • Let me call the original dataset Node 0.
  • I used classification method 1 to split Node 0 into Node 1 and Node 2.
    • Node 1 has low misclassification rate (Mostly consists of bad objects)
    • Node 2 has high misclassification rate (Roughly even mix of good and bad objects)
  • I now want to use classification method 2 to split Node 2 into Node 3 and 4

The "classification method" can be anything (LDA, QDA, SVM, CART, Random Forest, etc). So I guess what I am trying to achieve here is a "classification" tree (not CART), where each node is subjected to a different classification method to obtain an overall high "class purity". Basically, I want to use a mix of different classification methods to obtain reasonable results.

My problem lies in the loss of training data after the first split. I run out of usable data after I run it through "classification method 1", which was SVM in my case.

$\endgroup$
  • 1
    $\begingroup$ Maybe you should just try regular boosting? The effect should be the same, but the whole should be easier to validate. $\endgroup$ – user88 May 20 '11 at 20:02
  • $\begingroup$ @Jonathan, thanks for the edit. Could you tell us how big/small your training groups are? By "running out of usable data" do you mean that Node 2 contains few data points, hence that you don't have many Good objects to begin with? You could also consider general ensemble methods to combine classifications from different methods. $\endgroup$ – NRH May 20 '11 at 20:11
  • $\begingroup$ @mbq: Boosting sounds like a good option to explore. I am familiar with the idea, but have never implemented it before. I will have to investigate how to perform boosting in R. Any R packages or other resources that you may be able to recommend? $\endgroup$ – ialm May 20 '11 at 20:18
  • $\begingroup$ @NRH: The original dataset is approx 140,000 points. This is split into about 105,000 training set and 35,000 holdout set. From my edit, Node 0 starts with 35,000 entries, which is split into Node 1 with about 30,000 entries (99% "Bad") and Node 2 with about 5,000 entries (3400 "Bad", 2100 "Good"). $\endgroup$ – ialm May 20 '11 at 20:21
  • 1
    $\begingroup$ @Jonathan, keep the holdout as holdouts and train the second classifier on the training data again. That is, Node 0 is just your training data. It may or may not give improved performance, but that is how you should proceed. $\endgroup$ – NRH May 20 '11 at 21:28
3
$\begingroup$

Just to make sure that we are on the same page, I take it from your description that you consider a supervised learning problem where you know the Good/Bad status of your objects and where you have a vector of features for each object that you want to use to classify the object as either Good or Bad. Moreover, the result of training an SVM is to give a classifier, which, on the holdout data, gives almost no false Bad predictions, but 55% false Good predictions. I have not personally worked with problems with such a huge difference in error rates on the two groups. It suggests to me that the distribution of features in the two groups overlap, but that the distribution of features in the Bad group is more spread out. Like two Gaussian distributions with almost the same mean but larger variance for the group of Bad objects. If that is the case, I would imagine that it will be difficult, if not impossible, to improve much on the error rate for the Good predictions. There may be other explanations that I am not aware of.

Having said that, I think it is a sensible strategy to combine classification procedures in a hierarchical way as you suggest. First, one classifier splits the full training set into two groups, and then other classifiers split each of the groups into two groups etc. In fact, that is what classification trees do, but typically using very simple splits in each step. I see no formal problem in training whatever model you like on the training data that is classified as being Good by the SVM. You don't need to use the holdout data. In fact, you shouldn't, if you need the holdout data for assessment of the model.

Your second suggestion is closely related to just using the group classified as Good from your training data to train a second model. I don't see any particular reason to use CV-based classifications to obtain this group. Just remember, that if you are going to use CV, then the entire training procedure must be carried out each time.

My suggestion is to first get a better understanding of what the feature distributions look like in the two groups from low-dimensional projections and exploratory visualizations. It might shed some light on why the error rate on the Good classifications is so large.

$\endgroup$
  • $\begingroup$ Yes, we are on the same page, I am trying to create a hierarchical series of classification methods to try an reduce error rate. I have been looking at the distributions and densities of the features, and there is, as you describe, significant overlap in many of the features. I believe a potential problem is that the classification of "Good" and "Bad" in my dataset, and I will have to speak with subject matter experts to better determine (and hopefully quantify) what exactly distinguishes "good" from "bad". $\endgroup$ – ialm May 20 '11 at 20:13
  • $\begingroup$ @Jonathan, that is probably a good idea. The classification problem can also just be downright difficult. $\endgroup$ – NRH May 20 '11 at 20:30
6
$\begingroup$

I would use the same training dataset for both models, and use the same CV-folds for tuning. Don't use ANY of the 25% hold-out for training or tuning. Once you've fit your 2 models on the 75% training sample, evaluate your performance using the holdout.

If you are using R, the caret package has functions for creating folds on a dataset that you can re-use to tune multiple models and then evaluate their predictive accuracy. If you would like I can help you with some example code.

Edit: Here is the promised code, modified from the vignette for the package caret:

#Setup
rm(list = ls(all = TRUE)) #CLEAR WORKSPACE
set.seed(123)

#Pretend we only care about virginica
Data <- iris
virginica <- Data$Species=='virginica'
Data$Species <- NULL

#Look at the variable relationships
library(PerformanceAnalytics)
chart.Correlation(Data,col=ifelse(virginica,1,2))

#Create cross-validation folds to use for multiple models
#Use 10-fold CV, repeat 5 times
library(caret)
MyFolds <- createMultiFolds(virginica, k = 10, times = 5)
MyControl <- trainControl(method = "repeatedCV", index = MyFolds,
                summaryFunction = twoClassSummary,
                classProbs = TRUE)

#Define Equation for Models
fmla <- as.formula(paste("virginica ~ ", paste(names(Data), collapse= "+")))

#Fit some models
Data$virginica <- as.factor(ifelse(virginica,'Yes','No'))

svmModel <- train(fmla,Data,method='svmRadial',
    tuneLength=3,metric='ROC',trControl=MyControl)

rfModel <- train(fmla,Data,method='rf',
    tuneLength=3,metric='ROC',trControl=MyControl)

#Compare Models
resamps <- resamples(list(
    SVM = svmModel,
    RandomForest = rfModel
    ))
summary(resamps)
densityplot(resamps,auto.key = TRUE, metric='ROC')
$\endgroup$
  • $\begingroup$ Sorry, I think my question was a little unclear. I've made an edit clarify my thoughts! $\endgroup$ – ialm May 20 '11 at 20:04
  • 1
    $\begingroup$ @Jonathan check out the packages gbm, mboost, ada, and caTools for various boosting methods in R. $\endgroup$ – Zach May 20 '11 at 21:07
  • $\begingroup$ Thanks, I will take a look later today. I would also be very grateful for some example code, since you offered :) $\endgroup$ – ialm May 20 '11 at 21:23
  • $\begingroup$ @Jonathan I posted the example code, let me know if you need any help implementing it. $\endgroup$ – Zach May 24 '11 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.