2
$\begingroup$

After calculating overall loss, we backpropagate this loss through the student network. But we do not backpropagate through teacher network (I am not sure from that). And when a new batch arrives, we once again forward propagate the next batch through both two networks and so on.

My question is: do we backpropagate loss through the teacher or not? If yes, what is the logic of using a student network? Or teacher network is a network trained before and just used in inference mode while training student? Can anyone clarify this, please?

$\endgroup$

1 Answer 1

1
$\begingroup$

Or teacher network is a network trained before and just used in inference mode while training student?

Yes, exactly. The goal of knowledge distillation is to condense a complex 'teacher' model into a simpler 'student' one, with minimal degradation of performance. You train the student model on the teacher model's predictions, rather than the true labels. This doesn't require updating/altering the teacher model.

It is only to make lower the cost of the test (since we use a shallower model to test)?

Yes, that's right: you do this for faster or less computationally demanding inference. (There are other fringe reasons. Maybe you also want a more interpretable, shallower model, or maybe you have the fitted teacher model but either (a) no longer have the true labels, or (b) are trying to fit to a new dataset.)

$\endgroup$
4
  • $\begingroup$ Thank you so much for your response! So we can also use a teacher network trained for the same task using different inputs to train another student network (with the student's own inputs). $\endgroup$
    – Mas A
    Dec 15, 2021 at 14:09
  • $\begingroup$ You need the teacher's output for input $x_i$ to train the student on the same input $x_i$. $\endgroup$ Dec 15, 2021 at 15:00
  • $\begingroup$ I understand it. What I want to say is that: There are no restrictions on training student and teacher with the same input. I can train teacher with dataset A. Then train student network with dataset B by using the prediction results of teacher network (in test mode) for the same dataset B (in order to be able to compare their results.) $\endgroup$
    – Mas A
    Dec 15, 2021 at 15:12
  • $\begingroup$ Yes, exactly right. $\endgroup$ Dec 15, 2021 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.