How are text-to-speech systems' spectrogram frames aligned for loss calculation?

Question

A key aspect of how text-to-speech (TTS) machine-learning works is very unclear to me even after reading the Tacotron-2 paper and the Google AI blog.

https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html

https://arxiv.org/abs/1712.05884

How are spectrograms generated from the text are aligned with the ground-truth (GT) spectrograms? If each character in the input text leads to a fixed number of output spectrogram frames, then there is no guarantee (in fact, they will almost never be) that the number of output frames will be same as ground-truth frames or aligned. E.g. in the training-data, the speaker may speak a sentence faster or some words in the sentence faster, but the neural-net generates output spectrogram with a different spoken speed.

So how are spectrograms of tacotron-2 output and GT aligned? Does tacotron generate a fixed number of frames per input seq character or variable?

How does it know how many frame to generate for 2nd char 'E' in sentence "WE GO TO MOVIE" versus last char 'E'?

Answer 1

There is CTC loss which gets down when alignment gets better. So you run many iterations of training with CTC loss until it converge to good alignment. You can check something like https://distill.pub/2017/ctc/ for details although there many videos too.

Sometimes when annotation wrong and complex it doesn't converge, you can check the pictures here: https://github.com/espnet/espnet/issues/1360

Answer 2

Nikolay's answer is incorrect. For OP's tacotron2 paper, the outputs are aligned by teacher-forcing. This means the ground-truth output is fed as the pseudo input for the RNN. As the ground-truth itself is aligned, the model itself only has to predict the next time step while pretending all previous steps are perfectly aligned.