Working with the unlabeled documents from web for supervised text classification, even though the problem settings dictates using semi-supervised learning, I aim to compare several different supervised classifiers (e.g. Naive Bayes, classic SVM, and a deep learning one). Please consider the following empirical scenario:
Let say I have 10,000 unlabeled documents from which I can afford labeling only 1000 documents. And the rest 9000 are all unlabeled.
I can label the rest 9000 through defining a heuristic that works as a weak classifier by using keyword-matching.
The Goal: Comparing the accuracy of SVM, Naive Bayes and Deep Learning on the whole dataset.
For the sake of comparison, we need the ground-truth (gold standard) labels of the "whole" dataset. We may think of three empirical approaches to obtain the ground-truth:
1- Lebeling the rest of the dataset by using the non-accurate keyword-matching for the sake of measuring the final accuracy of each algorithm.
2- labeling the whole data first by using self-learning approach (which uses an internal weak classifier iteratively) and obtain a labeled dataset.
3- Using a semi-supervised algorithm (e.g. S3VM) for labeling the whole dataset.
Definitely, the human-labeled data is used in all of the above approaches.
Question: The problem with the first approach is that using the keyword-matching heuristic for the 9000 documents is not accurate enough to construct the gold standard labels. The problem with self learning approach is that it needs to use another internal classifier in each step, (Let say SVM). Then one can argue since SVM is used for obtaining the gold standard, then the comparison might be biased in favor of SVM eventually. And in case of using the third approach one can argue that, if we already assume that the best results (gold standard) is given by using a semi-supervise learning, why do we even bother with trying supervised learning?
Any thought on how to be able to obtain the ground truth in a more rigorous (standard) manner?