I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is that there is an oracle to label the documents.
Randomly select 500 documents and label them using an oracle, lets call this the hold out set H. Set H is never used for training purposes.
Select another 500 document set, Tr, label them using an oracle, train a SVM classifier and evaluate the accuracy of the resulting model on set H. Sample another 100 documents, get them labeled from the oracle, add them to Tr, train a SVM classifier and evaluate the accuracy on set H. Repeat the procedure until you reach the desired accuracy level and stop.
Personally I feel that a hold-out set so small may not be able to account for all the variation in the corpus in order to determine whether to stop the iterative learning process. But I am not able to quantify this. Would also appreciate some pointers to research papers.