9
$\begingroup$

I have a set of 400 labeled samples (8 numeric features) on which I trained a binary classifier.

The problem I am facing is that once the classifier is shipped to the users, I will get additional samples, but those will be unlabeled. I was researching common ways to incorporate unlabeled data in order to increase future classification accuracy. The way I see it I have 4 options:

  1. Forget about the existing binary classifier and use a semi-supervised learning algorithm such as S3VM

  2. Keep the existing binary classifier, use a transductive learning algorithm, such as label propagation, and use the newly (but possibly wrongly) labeled data to retrain the binary classifier; iterate this procedure.

  3. Keep the existing binary classifier, use a (supervised?) clustering algorithm to label new data, and use the newly (but possibly wrongly) labeled data to retrain the binary classifier; iterate this procedure. Maybe some mixture model with Expectation Maximization?

  4. Alternative idea?

While 3) seems rather flawed, because usual clustering algorithm optimize criterions different from labels I am not sure what to think about 1) and 2). What I do not like in 2) is that after we use a label propagation algorithm, we assume that these labels are correct and based on this new set of samples, we want to select new features and retrain our classifier. But a change in the missclassification rate now can stem form either a bad selection of features but might as well stem from the fact that the new labels are wrong. To me, 1) seems to reflect the situation the best. Am I understanding the situation correctly, i.e., is it true that 1) is superior to 2) and 2) is superior to 3)?

Or did I miss the point completely and an alternative approach is more appropriate than any of the 3?

$\endgroup$
3
  • $\begingroup$ You can use the existing classifier to get the estimates of the labels for the new unlabeled data. If your initial classifier is "good enough" and the new data comes from the same distribution (generated by the same process), it wouldn't be too difficult just to correct the mistakes. It should require much less effort to correct the mistakes than to annotate everything from scratch. This of course assumes that the person making corrections is an expert and can provide this information. $\endgroup$ May 18, 2015 at 20:44
  • $\begingroup$ @xeon Thank you for your comment, this is certainly a good idea. The existing classifier is decent, but not great and I am expecting to get a huge amount of data back. I am also interested in this setting from a theoretical point of view since it seems to me like a very common situation and I would like to know what other people think about it. $\endgroup$
    – user695652
    May 19, 2015 at 19:31
  • $\begingroup$ I am using this approach to train NER models for entity tagging in NLP. Whenever I see a misrecognized entity, instead of just copying the corrected example to the training set, I create similar examples and make the model recognize the test example. This way I am avoiding bias towards the test set. $\endgroup$ May 19, 2015 at 19:37

2 Answers 2

2
$\begingroup$

(3) doesn't have to be bad if you have some prior about what the clusters might look like, however you wouldn't be using your labelled data optimally. As you point out, you can iteratively train a classifier on its own output.

(2) isn't that different from (3) really, it'll depend on how good your metric is

(1) is what I would recommend, though it doesn't have to be S3VM. A Bayesian model would treat all the missing label as latent variables and learn the posterior distribution of both the missing labels and the classifier's parameters.

$\endgroup$
0
$\begingroup$

I would stick with the approach 1 and then use your set aside data to test if the new rule is any better than the original rule. It is not a given that this will be the case. Likely, you will use different class of functions for modeling than the one your original classifier used, and this may change the results for worse (as well as for better).

The approaches 2 and 3 seem doubtful because you add noise by using, possibly, wrong labels from your original classifier. It depends on the accuracy of your original classifier though.

The approaches 2 and 3 model your original classifier with new unlabeled data and different methods. The labels from a classifier are expected to be less reliable than the original training data. So, I do not see how these approaches can make you closer to the true labels.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.