Understanding semi supervised technique called mean teachers

Question

I am trying to understand applying semi supervised learning as described in this paper.

Describing the final recipe as described in this paper:

1. Take a supervised architecture and make a copy of it. Let's call the original model the student and the new one the teacher.

2. At each training step, use the same minibatch as inputs to both the student and the teacher but add random augmentation or noise to the inputs separately.

3. Add an additional consistency cost between the student and teacher outputs (after softmax).

4. Let the optimizer update the student weights normally.

5. Let the teacher weights be an exponential moving average (EMA) of the student weights. That is, after each training step, update the teacher weights a little bit toward the student weights.

I want to particularly understand this figure:

My understanding of it is:

We have two examples (DL1, DL2) (Large blue dots). Y-axis = predicted probability for class 2, X-axis = feature. Based on the threshold set, Blue half of the image is class 1 and pink half the class 2.

(a) The model (Ma) is learning to classify datapoints that are somewhere between the DL1 and DL2 as negative class which is something that we do not want

(b) We augmented the dataset by creating new points (small blue dots) by adding noise to DL1 and DL2. We assigned class 1 to these new datapoints and now when we train the model (Mb) it learns to be somewhat non variant around DL1 and DL2 in order to predict class 1 for the small blue dots

(c) An unlabeled example DU1 has entered the picture. Model (Mc1) is predicting it as class 1 but near the boundary between class 1 and class 2. Mc1 = The thin, pointed grey curve. Now, we augment the dataset by adding noise to unlabeled DU1 and create unlabeled datapoints (small black dots) and train the model (Mc2) with an additional L2 (or anything measuring consistency between two outputs) loss between the predictions of noisy unlabeled datapoints and DU1. Now, it learns to have a smooth boundary at the top of the curve rather than a pointed curve that it was learning earlier. Mc2 = thick grey curve. The DU1 is still predicted near to the prediction boundary. Why is this model better than the model in (b)?

(d) Did not get it. Please explain

(e) Did not get it. Please explain

Answer 1

I'll answer it for in case someone ever stumbles upon the need to understand the same issues. I mailed Antti Tarvainen, the author of the paper and this is what he responded:

Hi,

Thanks for the questions. I perhaps tried to pack too much into that figure so it may be hard to interpret our meaning. I will try to clarify.

(a) The model (Ma) is learning to classify datapoints that are somewhere between the DL1 and DL2 as negative class which is something that we do not want

Pretty much so. By chance, the model may have actually learned exactly the true function we are trying to model. (By true function I mean the actual real-world phenomenon we are trying to model. It's often called target function, but that would be confusing in this context.) But it is unlikely, because there is an infinite number of alternative functions that fit the data. Therefore, we want to regularize, i.e. make the model more likely to learn likely true functions. The following four subfigures attempt to explain how noise- and consistency-based regularization schemes such as mean teacher perform useful regularization.

(b) We augmented the dataset by creating new points (small blue dots) by adding noise to DL1 and DL2. We assigned class 1 to these new datapoints and now when we train the model (Mb) it learns to be somewhat non variant around DL1 and DL2 in order to predict class 1 for the small blue dots

Yes.

(c) An unlabeled example DU1 has entered the picture. Model (Mc1) is predicting it as class 1 but near the boundary between class 1 and class 2. Mc1 = The thin, pointed grey curve. Now, we augment the dataset by adding noise to unlabeled DU1 and create unlabeled datapoints (small black dots) and train the model (Mc2) with an additional L2 (or anything measuring consistency between two outputs) loss between the predictions of noisy unlabeled datapoints and DU1. Now, it learns to have a smooth boundary at the top of the curve rather than a pointed curve that it was learning earlier. Mc2 = thick grey curve. The DU1 is still predicted near to the prediction boundary. Why is this model better than the model in (b)?

Yes, that's correct. (Sorry for the very vague "gray curve" in the caption, and congratulations for deciphering it correctly. The thin curve is the teacher and the thick the student.)

I really should have drawn two figures like (c) (and (b) too): one where the model happens to be "lucky" and one where it happens to be "unlucky". As it is, I only drew an unlucky (c), which makes this confusing. Being unlucky motivates (d) and (e), but then (c) is indeed worse than (b).

The reason (c) is very unlucky is because it happens to peak right where the unlabeled example is. If it peaked at some other place, the noise around the unlabeled data point would smooth that peak, and bring it closer to the prediction of the unlabeled data point, and also the true value.

So sometimes a (c) is better than a (b), sometimes it is worse. We don't want to depend on luck, which motivates (d) and (e).

(d) Did not get it. Please explain

In (c) we were unlucky because we happened to pick the worst possible target. We don't want to be unlucky, so we sample the target prediction many times. We do this by adding noise to the teacher inputs too. Then the model learns to smooth the predictions around unlabeled data points towards the expected prediction in that neighborhood. The expected prediction is less noisy and probably a better target than any single prediction.

(e) Did not get it. Please explain

In (d) we reduced target noise by averaging over the neighborhood of each unlabeled data point (so in the space of input dimensions). But there's also noise in the model parameters. We can reduce our dependence on luck by averaging over model parameters. The way mean teacher does this is by averaging the model parameters over training steps.

(Arguably, dropout also adds noise to the model parameters, and thus is another way of improving expected model estimates. But in the paper we consider it a form of input noise, rather than parameter noise.)

I hope this clarifies it.