2
$\begingroup$

I'm reading about teacher forcing for neural translation applications here and here , but I am a little confused on the method. Why does teacher forcing speed up training? Also why in the Kaggle link are they only doing teacher forcing a percentage of the time?

$\endgroup$
2
  • $\begingroup$ Reading where? References might be good. ArXiv? Medium? Some random blog? YouTube? $\endgroup$ – EngrStudent Jan 8 at 20:23
  • 1
    $\begingroup$ Updated with my source. $\endgroup$ – Eisen Jan 8 at 20:34
3
$\begingroup$

Teacher forcing effectively means that instead of using the predictions of your neural network at time step t (i.e the output of your RNN), you are using the ground truth.

"Why does teacher forcing speed up training?"

Because if you don't use teacher forcing, it is autoregressive, meaning that you need to calculate the labels first before passing it at time step t.

"Also why in the Kaggle link are they only doing teacher forcing a percentage of the time?"

Because conditioning on the actual predictions might be more beneficial. Suppose that your RNN is unable to learn the input-output mapping to the desired precision. In that case, it is better to condition on its own faulty output so that it has a better chance of mitigating its own mistakes.

Related question: Is teacher forcing more accurate than using actual model output or just faster?

$\endgroup$
2
  • $\begingroup$ Hi! Thank you for your answer. What do you mean by "it is better to condition on its own faulty...better chance of mitigating its own mistakes."? Do you mean that we should not use teacher forcing 100% of the time so that we will have instances where mistakes are more likely, which is better for learning? $\endgroup$ – Eisen Jan 8 at 23:17
  • $\begingroup$ Yes, I mean that. There are good reasons to use teacher forcing, and I think in generic RNN training in PyTorch, it would be assumed that you are using teacher forcing because it is just faster. One way to look at is that you could have measurement error in your data, and the RNN functions like a filter trying to correct it. So if you want to introduce robustness to thee kind of measurement errors, it makes sense to sample both from the predicted and the observed samples. It is highly domain dependent whether it works but it can be treated as an additional hyperparameter. $\endgroup$ – boomkin Jan 9 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.