Interpolating speech from different speakers in trigger word detection

Question

Trigger word detection aims to essentially identify the timestamp of a "trigger word" occurring in a chunk of audio. The approaches I have seen online, in particular, following along with Andrew Ng's deep learning course, consist of having labeled audio data with a 1 for frames immediately proceeding the spoken trigger word and 0s everywhere else. Since there aren't any large datasets of this form, the general approach seems to rely upon generating these files by having 3 recording files; background clip (approx. 10 sec to add noise to audio), negative samples (approx. 1 sec of random spoken words) and positive samples (approx. 1 sec of spoken trigger word).

Negative samples and a positive sample are then overlayed (mutually exclusive of one another to avoid interference) onto the background clip and since we know the lengths of all audio files, we already know the timestamp of the trigger word - it's wherever we have decided to interpolate it.

My first question is this: why wouldn't the approach just be to "pattern match" the waveform of the trigger word (maybe an average over many speakers) with the waveform of the audio file? Or, similar to speech recognition, why not just identify the phonemes in the audio and see if they match the phonemes constituting the trigger word.

My second question is this: in generating the audio files with the aformentioned procedure, is it detrimental to use negative examples and a positive example/s from different speakers in the same audio clip? My concern would be that the network might learn a relationship between new speaker and trigger word not trigger word waveform and trigger word since there will be a correlation between the trigger word and the fact that is always occurs with a new speaker. Is it perhaps smart to try and break this correlation by using different speakers in the audio between negative samples, in the hope that the network will realise that there are times when there is a change in speaker, but not the presence of the trigger word? Or would is this likely too difficult a feature for the network to learn anyway?

Thanks!

Answer 1

My first question is this: why wouldn't the approach just be to "pattern match" the waveform of the trigger word (maybe an average over many speakers) with the waveform of the audio file?

This is a valid approach, just not very accurate practically. The waveforms are pretty diverse - wide pitch variations, timing variations, etc. It is hard to match them quickly and accurately, although this method was actively used in 70's when computers were slow.

Or, similar to speech recognition, why not just identify the phonemes in the audio and see if they match the phonemes constituting the trigger word.

This is a valid approach too, however, phonemes are also not very clearly articulated by many people, so it is hard to match them. Phoneme recognition accuracy for clear audio is 15% (state of the art these days). For noisy audio it is even less. This was the popular approach in 90's

My second question is this: in generating the audio files with the aformentioned procedure, is it detrimental to use negative examples and a positive example/s from different speakers in the same audio clip? My concern would be that the network might learn a relationship between new speaker and trigger word not trigger word waveform and trigger word since there will be a correlation between the trigger word and the fact that is always occurs with a new speaker. Is it perhaps smart to try and break this correlation by using different speakers in the audio between negative samples, in the hope that the network will realise that there are times when there is a change in speaker, but not the presence of the trigger word? Or would is this likely too difficult a feature for the network to learn anyway?

Network will likely not learn speaker change this way, but more variety in the training audio should definitely help.