Semi-supervised Learning Training

Question

I have got some data partially labelled. Therefore, I would like to apply semi-supervised learning for this dataset. Basically, I trained the Support Vector Machine (SVM) using the data with labels and build the classifier and tune the SVM parameters. Then I do the prediction for the unlabelled data. Here I have got some questions:

1. After I have done the training, I have obtained a training accuracy of 96.85%, and the confusion matrix is

Now we can see that the training have got some mis-classification. And should I do the prediction directly now or should I remove the mis-classified cases and then do the prediction for the unlabelled data? By remove the mis-classified data, and then do the training again I can get 100% training accuracy.

2. If I don't need to remove the mis-classified data during the training. What are the benefits of this? I think by including the mis-classified cases can increase the variability of the model, am I right?

Answer 1

Don't remove the misclassified examples, as they give information about the correct location of the decision boundary. The decision surface for an SVM is defined by the support vectors, i.e. those patterns that lie on or within the margins, so if you selectively delete them, none of the theory underpinning the SVM will be valid. This can be a useful thing to do in order to reduce the run time expense of the classifier (or alternatively flipping the labels of misclassified patterns, see this paper), but I wouldn't expect it to be a good idea from the perspective of generalisation.

Also don't pay too much attention to the training set accuracy, it isn't much of a useful indicator of the quality of the classifier as classifiers that have over-fitted the training data will have an unrealistically optimistic training set error.

For semi-supervised learning (using self-training), it may be better to only use those unlabelled patterns where the output of the SVM has an absolute magnitude greater than one (i.e. the SVM is confident of the classification, rather than being a bit uncertain).

Answer 2

Removing data shouldn't be performed a posteriori after getting the accuracy results. You can remove some observations only after a noise or outlier analysis. The model you got should be accurate enough to be applied directly on the unlabelled data. Thus you'll avoid overfitting and increase the variability as you said.