I am trying to understand applying semi supervised learning as described in this paper.
Describing the final recipe as described in this paper:
Take a supervised architecture and make a copy of it. Let's call the original model the student and the new one the teacher.
At each training step, use the same minibatch as inputs to both the student and the teacher but add random augmentation or noise to the inputs separately.
Add an additional consistency cost between the student and teacher outputs (after softmax).
Let the optimizer update the student weights normally.
Let the teacher weights be an exponential moving average (EMA) of the student weights. That is, after each training step, update the teacher weights a little bit toward the student weights.
I want to particularly understand this figure:
My understanding of it is:
We have two examples (DL1, DL2) (Large blue dots). Y-axis = predicted probability for class 2, X-axis = feature. Based on the threshold set, Blue half of the image is class 1 and pink half the class 2.
(a) The model (Ma) is learning to classify datapoints that are somewhere between the DL1 and DL2 as negative class which is something that we do not want
(b) We augmented the dataset by creating new points (small blue dots) by adding noise to DL1 and DL2. We assigned class 1 to these new datapoints and now when we train the model (Mb) it learns to be somewhat non variant around DL1 and DL2 in order to predict class 1 for the small blue dots
(c) An unlabeled example DU1 has entered the picture. Model (Mc1) is predicting it as class 1 but near the boundary between class 1 and class 2. Mc1 = The thin, pointed grey curve. Now, we augment the dataset by adding noise to unlabeled DU1 and create unlabeled datapoints (small black dots) and train the model (Mc2) with an additional L2 (or anything measuring consistency between two outputs) loss between the predictions of noisy unlabeled datapoints and DU1. Now, it learns to have a smooth boundary at the top of the curve rather than a pointed curve that it was learning earlier. Mc2 = thick grey curve. The DU1 is still predicted near to the prediction boundary. Why is this model better than the model in (b)?
(d) Did not get it. Please explain
(e) Did not get it. Please explain