# Binary classifier - dividing dataset into training and evaluation sets

I have a Hidden Markov Model for binary classification and two datasets:

• positive instances
• negative instances (way more data than the positive ones)

In order to evaluate the performance of the model I did the following:

1. Do leave one out cross validation over the positive instances. Basically remove an instance from the positive set, train over the rest, then evaluate the instance I removed and saved the result; Repeat for each instance.
2. Train over all the positive instances and then evaluate each negative instance. Save results
3. Plot ROC curve with the data from 1 and 2.

This approach is pretty time intensive as I have to train my model N+1 times where N equals the number of positive instances.

Someone suggested that I combine both data sets and then divide them:

• 2/3 training set
• 1/3 evaluation set

and maintain in both sets the same percentage of positive/negative instances.

Maybe I understood something wrong but I am a bit confused as to how this helps exactly when I have negative instances in the training data!?

Wouldn't that negatively bias my classifier when evaluating over the remaining 1/3 instances? Moreover, I would also get less data points for the ROC curve?

Can anybody help clarify the approach or suggest a better one?

-
I don't think with the imbalance you have in positive versus negative instances that you should throw out information on negative cases bycutting them down to be the same as much lower total of positive instances. –  Michael Chernick Jul 7 '12 at 3:46
I don't understand how you are training the model to correctly identify negative instances. If you are doing some training in step 2 when you say "evaluate," it still sounds like you are training $N$ or $N+1$ times on each positive for each negative. This can be a bad thing to do. The trivial classifier which decides everything is positive may perform quite well compared with classifiers which are right on over $90\%$ of both positive and negative points. It can be ok to oversample the positive instances, but the weight on positive points should depend on #positive/#negative, not #positive. –  Douglas Zare Jul 7 '12 at 9:35
Regarding the training: for the positive instances I trained on N-1 instances and evaluated the Nth (repeat N times for each positive instance). For the negatives instances: train once on all the positive instances and evaluate the negative instances one by one(here I only train the model once). When I say evaluate I use the Forward Algorithm to get the P(instance | Model), that is, the probability of an instance/sequence given the model. –  Morat Jul 7 '12 at 11:11