I'm reading about teacher forcing for neural translation applications here and here , but I am a little confused on the method. Why does teacher forcing speed up training? Also why in the Kaggle link are they only doing teacher forcing a percentage of the time?
2
-
$\begingroup$ Reading where? References might be good. ArXiv? Medium? Some random blog? YouTube? $\endgroup$– EngrStudentCommented Jan 8, 2021 at 20:23
-
1$\begingroup$ Updated with my source. $\endgroup$– EisenCommented Jan 8, 2021 at 20:34
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Teacher forcing effectively means that instead of using the predictions of your neural network at time step t (i.e the output of your RNN), you are using the ground truth.
"Why does teacher forcing speed up training?"
Because if you don't use teacher forcing, it is autoregressive, meaning that you need to calculate the labels first before passing it at time step t.
"Also why in the Kaggle link are they only doing teacher forcing a percentage of the time?"
Because conditioning on the actual predictions might be more beneficial. Suppose that your RNN is unable to learn the input-output mapping to the desired precision. In that case, it is better to condition on its own faulty output so that it has a better chance of mitigating its own mistakes.
Related question: Is teacher forcing more accurate than using actual model output or just faster?
-
$\begingroup$ Hi! Thank you for your answer. What do you mean by "it is better to condition on its own faulty...better chance of mitigating its own mistakes."? Do you mean that we should not use teacher forcing 100% of the time so that we will have instances where mistakes are more likely, which is better for learning? $\endgroup$– EisenCommented Jan 8, 2021 at 23:17
-
$\begingroup$ Yes, I mean that. There are good reasons to use teacher forcing, and I think in generic RNN training in PyTorch, it would be assumed that you are using teacher forcing because it is just faster. One way to look at is that you could have measurement error in your data, and the RNN functions like a filter trying to correct it. So if you want to introduce robustness to thee kind of measurement errors, it makes sense to sample both from the predicted and the observed samples. It is highly domain dependent whether it works but it can be treated as an additional hyperparameter. $\endgroup$– boomkinCommented Jan 9, 2021 at 11:11