2
$\begingroup$

I am currently reading the paper Distilling the Knowledge in a Neural Network and in the introduction I came across the following sentence -

When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases, so the small model can often be trained on much less data than the original cumbersome model and using a much higher learning rate.

  1. If the soft targets have high entropy for some training cases, why would this lead to less variance in the gradient updates for those training cases?
  2. How did the authors conclude that the small model can be trained on much less data (in comparison to the cumbersome model's training data) based on the first point?
Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

Welcome to CV, GaneshTata.

Hinton is a legend. Reading the papers of the top 100 ML authors is a great exercise.

The way I took this was the difference between nominal levels versus the probability for one-hot encoding of that nominal/factor. In binomial the true-false is a one-bit information source while the float describing it can have 32bits of informative value. After you get the class membership probabilities behaving well you can argmax to determine class.

For example:
Imagenet has 1000 classes. If someone were to target a learner's output to be a single categorical/hard value representing the class index, then for each class the answers are only true or false. A better way to do this is to go one step inside, and look at the probability of membership before the arg-max. Instead of getting a single boolean to inform the training for a single class, there would be 1000 continuous values between 0 and 1 informing class membership. Personally I see a connection between this paper, and the zero-shot learning paper

The soft values are probabilities between 0 and 1 so they don't tend to get runaway gradients, the square of the slope of which is the variance from delta method.

If there is lower variance, then the epochs are more about moving forward than pushing against a river of noise. All else being equal, less noisy data is easier to learn. If you get to the destination quicker then you can use fewer epochs to get to the same error, aka fewer training steps.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the wonderful explanation! Just to clarify, when you say "they don't tend to get runaway gradients", do you mean that the gradients are stable (or fluctuate lesser)? $\endgroup$
    – GaneshTata
    Nov 18, 2021 at 18:30
  • 1
    $\begingroup$ For NN based learners they predict, compute the error, then they backpropagate error and use it to update the weights of the network. If you predict using volume of stock markets, your targets are on the order of hundreds of millions. This means you might compute a jacobian (mulitvariate slope) suggesting you move weights around by a factor of millions. This can be a bad thing. Also, for the probabilities there is correlation: In general when one is high others are low. Hinton finds "3" in that paper. :) $\endgroup$
    – EngrStudent
    Nov 18, 2021 at 18:34
  • $\begingroup$ That makes sense, thank you so much! $\endgroup$
    – GaneshTata
    Nov 18, 2021 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.