0
$\begingroup$

I'm programming a LSMT network from scratch (I'm interested in understandind their internal functioning), but my question applies to any RNN. There is somehting which I'm not sure to be doing right. Let's say I'm training my network with sequences $s_i$, each of length $l_i$ and with label $c_i$. This is what I'm doing, for each $s_i$:

  1. Reset the network internal state.
  2. Feed each element $s_i(1)$, $s_i(2)$, ..., $s_i(l_i-1)$, completely disregaring the output of the network.
  3. When the last element of the sequence, i.e. $s_i(l_i)$, comes along, I compare the output of the network with the class of the sequence, and apply BPTT.

Is this approach correct? It does make sense to me, but when I try to apply it to a slightly different situation I get confused.

Let's say that now I'm doing speech processing. Each sequence $s_i$ now comprises $n_i$ phonemes. For $n_i = 1$ I could of course apply the previous approach. But for greater numbers I'm not sure what I should do.

$\endgroup$
1
$\begingroup$

The output of an rnn is not a 'class'.

You'll need to do something to the output to make it into a class. typically this will involve adding something like:

  • a fully-connected layer, with the number of output neurons identical to the number of classes
  • a softmax layer, to normalize the outputs of the fully-connected layer into a probability distribution, ie sum to 1
  • finally, an argmax, which chooses the class to be the neuron of the softmax having the maximum value

After adding these layers onto the output of your rnn, you can then backpropagate your target classes through these as normal, and thence into the rnn.

By the way the input to this sequence of output layers will be the final state of your rnn, since you're doing sequence -> one essentially, squashing your sequence of input values into a single feature vector, being the final state of your rnn.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.