For my classification problem, I am trying to classify an object as Good or Bad. I have been able to create a good first classification step that separates the data into 2 groups using SVM.
After tuning the parameters for the SVM using a training/holdout set (75% training, 25% holdout), I obtained the following results from the holdout set: Group 1 (model classified as Bad) consisted of 99% Bad objects, and Group 2 (model classified as Good) consisted of about 45% Good objects and 55% Bad objects. I verified the performance of the model using k-fold CV (k=5) and found the model to be stable and perform relatively consistently in terms of misclassification rates.
Now, I want to pass these objects through another round of classification by training another model (may or may not be SVM) on my group 2 of maybe good/maybe bad objects to try and correctly classify this second group now that I have gotten rid of the obviously bad objects.
I had a couple of thoughts, but am unsure of how to proceed.
(1) My first idea was to use the data from the classified objects from the Holdout set to train another model. I was able to train another classification model from the results of the holdout set. The problem is I am using less than 25% of the original data, and I am worried of overfitting on a very small subset of my data.
(2) My second idea was to gather the results of the 5-fold CV to create another dataset. My reasoning is that since the data is partitioned into 5 parts, and each part is classified into two groups from a model trained by the other 4 parts, I thought that I could aggregate the predicted results of the 5 parts to obtain a classified version of my original dataset and continue from there.
The only problem is, I have a sinking feeling that both methods are no good. Could CV shed some light on some possible next steps?
EDIT
Sorry, my question was badly worded. Let me try to clarify what I am trying to do. It can be thought of like a tree...
- Let me call the original dataset Node 0.
- I used classification method 1 to split Node 0 into Node 1 and Node 2.
- Node 1 has low misclassification rate (Mostly consists of bad objects)
- Node 2 has high misclassification rate (Roughly even mix of good and bad objects)
- I now want to use classification method 2 to split Node 2 into Node 3 and 4
The "classification method" can be anything (LDA, QDA, SVM, CART, Random Forest, etc). So I guess what I am trying to achieve here is a "classification" tree (not CART), where each node is subjected to a different classification method to obtain an overall high "class purity". Basically, I want to use a mix of different classification methods to obtain reasonable results.
My problem lies in the loss of training data after the first split. I run out of usable data after I run it through "classification method 1", which was SVM in my case.