I was reading the report of the winning solution of a Kaggle competition (Malware Classification). The report can be found in this forum post. The problem was a classification problem (nine classes, the metric was the logarithmic loss) with 10000 elements in the train set, 10000 elements in the test set.
During the competition, the models were evaluated against 30% of the test set. Another important element is that the models were performing very well (close to 100% accuracy)
The authors used the following technique:
Another important technique we come up is Semisupervised Learning. We first generate pseudo labels of test set by choosing the max probability of our best model. Then we predict the test set again in a cross validation fashion with both train data and test data. For example, the test data set is split to 4 part A, B, C and D. We use the entire training data, and test data A, B, C with their pseudo labels, together as the new training set and we predict test set D.
The same method is used to predict A, B and C. This approach, invented by Xiaozhou, works surprisingly well and it reduces local cross validation loss, public LB loss and private LB loss. The best Semisupervised learning model can achieve 0.0023 in private LB log loss, which is the best score over all of our solutions.
I really don't see how it can improve the results. Is it because 30% of the test set was "leaked" and it was a way to use this information ?
Or is there any theoretical reason explaining why it works ?
