0
$\begingroup$

I have to make a live speech recognition program that can spot specific words in a spoken sentence. For now I have to recognize the words "yes" and "no".

I already trained Google's model and it worked, but only if the words were spoken apart. (The dataset they provide contains wav files of 1 second each, recorded at 16kHz in a PCM 16bit format of words spoken separately).

Since I want to spot the words in a sentence, I'm not sure the model they provide (cnn) is suited for this use case (is it?). I'm thinking about something more like LSTMs. So I have few questions:

  1. Is there a way to do it "simply" using convnets or something similar?
  2. Is LSTM the best suited network for this task?
  3. If yes, is it possible to use Google's dataset of separately spoken words to train a LSTM network?
  4. If not, what kind of network should I use?

Any answer, even if not complete, would be greatly appreciated! Thanks!

EDIT: I realized that my post could be confusing, but really what I want to know is if google's dataset of separately spoken words is enough to train a network to recognize spoken words within a sentence. And if yes, what kind of network should I use ?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ Before LSTM's and Conv Nets, this problem was tackled using Hidden Markov Models. I suggest you expand your research in that direction as well. $\endgroup$
    – Zhubarb
    Feb 5, 2018 at 9:51

1 Answer 1

Reset to default
1
$\begingroup$

Although I am no expert on NLP, I believe you could use a fully convolutional network followed by a max-pooling operation. The big advantage of FCNs is that they can work with arbitrary input sizes, so it does not matter how long the input sentence is.

I don't know the details of the network architecture you are referencing, but I suppose it contains some conv layers followed by a fully connected ones. In that case, you can use the conv layers you already have and then add a 1x1x4 conv-layer (the kernels are representing classes 'contains-yes', 'does-not-contain-yes', 'contains-no', 'does-not-contain-no') and some global max pooling aggregating the information from the whole sentence. Then using a separate softmax on 'yes' channels and 'no' channels, you can get a probability that the sentence contains 'yes' and 'no'.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for these information ! I didn't know about FCNs ! I'm definitely going to look at this, I'll keep you updated! $\endgroup$
    – Omar Aflak
    Feb 5, 2018 at 16:05
  • $\begingroup$ I'm still wondering, is it going to work even if the dataset is just words and not sentences containing these words ? (I think this is the problem) $\endgroup$
    – Omar Aflak
    Feb 5, 2018 at 16:25
  • $\begingroup$ Let's say you have a convolution layers that can extract useful information from a word (decide if it is yes or no). Now instead of using them on a single word, they are used in a sliding window fashion on the whole sentence, yielding high response in the parts of the sentence containing the relevant word. By global max pooling you aggregate this information from the whole sentence. The length of the sentence does not matter as long as the basic building block (the conv layers) work well. $\endgroup$
    – Jan Kukacka
    Feb 5, 2018 at 20:46
  • $\begingroup$ Okay I see. But how is it different from a regular CNN which is fed with several small frames in time ? $\endgroup$
    – Omar Aflak
    Feb 5, 2018 at 20:54
  • $\begingroup$ It is almost the same. Instead of you having to implement everything yourself, FCN works like this automatically. I suppose you were asking how to extend your existing CNN to analyze whole sentences. $\endgroup$
    – Jan Kukacka
    Feb 5, 2018 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.