I have a Hidden Markov Model for binary classification and two datasets:
- positive instances
- negative instances (way more data than the positive ones)
In order to evaluate the performance of the model I did the following:
- Do leave one out cross validation over the positive instances. Basically remove an instance from the positive set, train over the rest, then evaluate the instance I removed and saved the result; Repeat for each instance.
- Train over all the positive instances and then evaluate each negative instance. Save results
- Plot ROC curve with the data from 1 and 2.
This approach is pretty time intensive as I have to train my model N+1 times where N equals the number of positive instances.
Someone suggested that I combine both data sets and then divide them:
- 2/3 training set
- 1/3 evaluation set
and maintain in both sets the same percentage of positive/negative instances.
Maybe I understood something wrong but I am a bit confused as to how this helps exactly when I have negative instances in the training data!?
Wouldn't that negatively bias my classifier when evaluating over the remaining 1/3 instances? Moreover, I would also get less data points for the ROC curve?
Can anybody help clarify the approach or suggest a better one?