Does speech recognition model training require transcript timestamps

Question

I don't quite understand how a recurrent neural network or LSTM is trained for automatic speech transcription. Say I have n audio files of speech, each with an associated transcript. I get that the neural network tries to predict the current phoneme, but how does the network know whether it was correct in guessing that phoneme or not? That phoneme occurred at a certain timestamp in the audio, does that mean that the transcript file needs to contain the timestamp of each phoneme for training?

UPDATE

Regarding the architecture of a recurrent neural network for this task, say the network's goal is to predict the word that was spoken. Does the the network just have a single output which is a word vector and then we train it based off of the ground-truth word vector? Is it possible to output after each frame the probability that that frame belongs to a certain phoneme? If so, how would one train the network without timestamp information?

For example, in this article, a predicted letter is outputted for each frame. Surely there wasn't any manual tagging for each frame for all the training files...

Answer 1

Timestamps are not required, network learns to align during training by itself.

Neural network starts with random values first and then over training it improves loss function more and more which corresponds to proper timecodes. It learns to insert blank tokens into appropriate frames so that loss goes down the model become better. With best loss function phonemes are properly identified.

For more details you can check

https://towardsdatascience.com/intuitively-understanding-connectionist-temporal-classification-3797e43a86c

In any case, the key idea to learn is CTC loss.

Answer 2

Different training setups are possible, mostly depending on data availability. One can have phoneme level labeling, then the time of each phoneme is known. However such detailed labels are incredibly time consuming and expensive to aquire. More often the network will be set up to predict an entire word or sentence, perhaps 0.5 to 2 seconds of audio. This requires no phoneme information at all, just the transcribed text.

The network still processes audio at the granularity of phonemes (50 ms MFCC frames or similar), but aggregated over the entire sentence (maybe 40 such frames).