Soft Target in Knowledge Distillation I am currently reading the paper Distilling the Knowledge in a Neural Network and in the introduction I came across the following sentence -

When the soft targets have high entropy, they provide much more
information per training case than hard targets and much less variance
in the gradient between training cases, so the small model can often
be trained on much less data than the original cumbersome model and
using a much higher learning rate.


*

*If the soft targets have high entropy for some training cases, why would this lead to less variance in the gradient updates for those training cases?

*How did the authors conclude that the small model can be trained on much less data (in comparison to the cumbersome model's training data) based on the first point?

 A: Welcome to CV, GaneshTata.
Hinton is a legend.  Reading the papers of the top 100 ML authors is a great exercise.
The way I took this was the difference between nominal levels versus the probability for one-hot encoding of that nominal/factor.  In binomial the true-false is a one-bit information source while the float describing it can have 32bits of informative value.  After you get the class membership probabilities behaving well you can argmax to determine class.
For example:
Imagenet has 1000 classes.  If someone were to target a learner's output to be a single categorical/hard value representing the class index, then for each class the answers are only true or false.  A better way to do this is to go one step inside, and look at the probability of membership before the arg-max.  Instead of getting a single boolean to inform the training for a single class, there would be 1000 continuous values between 0 and 1 informing class membership.  Personally I see a connection between this paper, and the zero-shot learning paper
The soft values are probabilities between 0 and 1 so they don't tend to get runaway gradients, the square of the slope of which is the variance from delta method.
If there is lower variance, then the epochs are more about moving forward than pushing against a river of noise.  All else being equal, less noisy data is easier to learn.  If you get to the destination quicker then you can use fewer epochs to get to the same error, aka fewer training steps.
