0
$\begingroup$

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself.
However, it seems I have not understood some parts as it was not covered or explained in the course. Right now, I don't know if my implementation is wrong or some it is some hyper parameter tuning issue since when trying a simple example, I get weird error in which very soon, all entries reach 1 (as the confidence of a specific class!). I explain shortly.
But before I get to that, I want to reiterate my understanding of how backward phase is carried out in a simple RNN layer.

In forward pass we simply calculate two terms:

$h_t = tanh(W_1.Xt + W_2.h_{t-1} + b_h)$
$output_t = softmax(W3.h_t + b_o)$

Although in practice the dimensions are different and instead the terms look like this :

$h_t = tanh(Xt.W_1 + h_{t-1}.W_2 + b_h)$
$output_t = softmax(h_t.W3.T + b_o)$

This is the python implementation that I wrote :

def rnn_cell_foward(self, xt, h0):
    """
        Run the forward pass for a single timestep of a vanilla RNN that uses a tanh
        activation function.
        The input data has dimension D(vocabsize in case we have nlp use), the hidden state has dimension H, and we use
        a minibatch size of N.
        Inputs:
        - x: Input data for this timestep, of shape (Batch_size, vocabsize_or_basically_input_dim_size).
        - h0: Hidden state from previous timestep, of shape (Batch_size, HiddenSize)
        - W1: Weight matrix for input-to-hidden connections, of shape (vocabsize, HiddenSize)
        - W2: Weight matrix for hidden-to-hidden connections, of shape (HiddenSize, HiddenSize)
        - W3: Weight matrix for hiddent-to-output connections, of shape(vocabsize_or_output_dim_size, hidden_state_size)
        - bh: Biases of shape (HiddenSize,)
        - bo: Biases of shape (vocabsize_or_output_dim_size,)

        Returns a tuple of:
        - next_h: Next hidden state, of shape (Bachsize, HiddenSize)
        - output: output
   """

    h_t = np.tanh(np.dot(xt, self.W1) + np.dot(h0, self.W2) + self.bh)
    o_t = softmax(np.dot(h_t, self.W3.T) + self.bo)
    return o_t, h_t

def rnn_layer_forward(self, Xt):

    batch, input_dim_size, T_x = Xt.shape
    # allocate H, and O for the first time
    if (self.H is None and self.O is None):
        print('first time call!')
        self.H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
        self.O = np.zeros(shape=(batch, self.outputsize, T_x))

    if (self.mode == 1):
        h_previous = np.zeros(shape=(batch, self.hidden_state_size))
    else:
        h_previous = self.H[:, :, -1]

    for t in range(T_x):
        output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous) 
        self.H[:, :, t] = h_t
        self.O[:, :, t] = output

        # Our current/new hiddenstate will be the previous hiddenstate for the next round,
        h_previous = h_t

    return self.O, self.H

So far so good. Now for the Back-propagation, we need to do the following :

  1. Calculate the error by subtracting the output from the label (e = output - label)

  2. Calculate the gradients for the output term

  3. Calculate the gradients for the hidden state term.
  4. For this we start from the last time-step and loop back to the start.

Now this is where my confusion begins. first of all, how am I supposed to subtract the output? do I simple do :

e = output - label

or should I specifically subtract the entry equal to the current time step? i.e :

e = output[t] - label[t]

(especially when we are dealing with sequence to sequence rnn, when we have a sentence/word as input, and we want to generate a sentence/word as the output)

If I go the first route, $e$ will be a vector of size $output$. and for loss, I need to sum all of the entries. Is this expected behavior?
If I go the second route, $e$ will be a single number, I can use it for the loss, but is it the right way to do so? Since if we go the second route, very quickly our network would learn just to set each entry to 1 regardless of other entries and that is not good. in fact this is what I'm getting now I guess. I have seen this implementation in Andrew Ng's implementation and that's why its confusing me! for him it works at least, but mine it is not! and also it doesn't make any sense.
When we use a batch of samples instead of 1, what should I do then? in the first route, I'd just sum everything, do I need to sum all errors here as well?

This error is important because it will be multiplied by the gradients of the output term (i.e $dW_3, dh_t, db_o$)

I tried both scenarios and both failed. Below is the methods I wrote for Back-propagation :

 def rnn_cell_backward(self, xt, h, output, true_label, h_gradient_or_dh, t):
    """
    h: is the next state 
    output: is the output of the current cell!
    true_label: is the true label for the current output
    h_gradient_or_dh: is the dh which in the beginning is zero
                     and is updated as we go backward in the backprogagation
                     remember the backward pass is essentially a loop! 
                     so have that in mind and you will be fine!
    """

    e = output
    e[0,t] = output[0,t] - true_label[0,t]
    per_ts_loss = output[0,t] - true_label[0,t]

    # must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
    dW3 = np.dot(e.T, h)
    dh = np.dot(e, self.W3) + h_gradient_or_dh  # from later cell
    # dbo +=e.1, we use sum, becasue we have a batch
    #e is a vector, when it is subtracted from label, the result will be added to dbo
    dbo = np.sum(e, axis=0) #e.reshape(e.shape[1]) #np.sum(e, axis=0)
    # print('e=',e)

    # the input part
    dtanh = (1 - h**2) * dh

    # compute the gradient of the loss with respect to W1
    dxt = np.dot(dtanh, self.W1.T)
    # must have the shape of (vocab_size, hidden_state_size)
    dW1 = np.dot(xt.T, dtanh)

    # compute the gradient with respect to W2
    dh_prev = np.dot(dtanh, self.W2)
    # shape must be (HiddenSize, HiddenSize)
    dW2 = np.dot(dh_prev.T, dtanh)

    #print('dtanh.shape = ',dtanh.shape)
    # dbh += dtanh.1, we use sum, since we have a batch
    dbh = np.sum(dtanh, axis=0)

    return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss

def rnn_layer_backward(self, Xt, labels, H, O):

    dW1 = np.zeros_like(self.W1)
    dW2 = np.zeros_like(self.W2)
    dW3 = np.zeros_like(self.W3)
    # must have the shape(batch,hiddensize)
    dBh = np.zeros_like(self.bh)
    # must have the shape(batch,outputsize)
    dBo = np.zeros_like(self.bo)
    dH_prev = np.zeros_like(H[:, :, 0])
    dh = np.zeros_like(H[:, :, 0])
    dXt = np.zeros_like(Xt)

    _, _, T_x = Xt.shape
    loss = 0

    for t in reversed(range(T_x)):
        dw1, dw2, dw3, dbh, dbo, dxt, dh_prev, dh, e = self.rnn_cell_backward(
                                                                            Xt[:, :, t],
                                                                             H[:, :, t],
                                                                             O[:, :, t],
                                                                        labels[:, :, t],
                                                                        dh, t)
        dW1 += dw1
        dW2 += dw2
        dW3 += dw3
        dBh += dbh
        dBo += dbo
        dH_prev += dH_prev
        # Update the loss by subtracting the cross-entropy term of this time-step from it.
        loss -= np.log(e)#e.sum())

    return dW1, dW2, dW3, dbh, dBo, dXt, dH_prev, dh, loss

Now for the test, I tried the Andrew Ng's dinosaur name generation example. but at the very beginning I get this weird error where all outputs are 1!

Any help is greatly appreciated.
By the way this is my simple rnn definition:

import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)


class RNNClass(object):
    def __init__(self, vocab_size, outputsize, hidden_state_size=100, mode=1):

        np.random.seed(1)
        self.W1 = np.random.randn(vocab_size, hidden_state_size) * 0.01
        self.W2 = np.random.randn(hidden_state_size, hidden_state_size) * 0.01
        self.W3 = np.random.randn(outputsize, hidden_state_size) * 0.01
        self.bh = np.random.randn(hidden_state_size)
        self.bo = np.random.randn(outputsize)

        self.outputsize = outputsize
        self.hidden_state_size = hidden_state_size
        self.H = None
        self.O = None
        self.mode = mode

        print('W1: ', self.W1.shape)
        print('W2: ', self.W2.shape)
        print('W3: ', self.W3.shape)
        print('bh: ', self.bh.shape)
        print('bo: ', self.bo.shape)

    import numpy as np

    def rnn_cell_foward(self, xt, h0):
        """
            Run the forward pass for a single timestep of a vanilla RNN that uses a tanh
            activation function.
            The input data has dimension D(vocabsize in case we have nlp use), the hidden state has dimension H, and we use
            a minibatch size of N.
            Inputs:
            - x: Input data for this timestep, of shape (Batch_size, vocabsize_or_basically_input_dim_size).
            - h0: Hidden state from previous timestep, of shape (Batch_size, HiddenSize)
            - W1: Weight matrix for input-to-hidden connections, of shape (vocabsize, HiddenSize)
            - W2: Weight matrix for hidden-to-hidden connections, of shape (HiddenSize, HiddenSize)
            - W3: Weight matrix for hiddent-to-output connections, of shape(vocabsize_or_output_dim_size, hidden_state_size)
            - bh: Biases of shape (HiddenSize,)
            - bo: Biases of shape (vocabsize_or_output_dim_size,)

            Returns a tuple of:
            - next_h: Next hidden state, of shape (Bachsize, HiddenSize)
            - output: output
       """

        h_t = np.tanh(np.dot(xt, self.W1) + np.dot(h0, self.W2) + self.bh)
        o_t = softmax(np.dot(h_t, self.W3.T) + self.bo)
        return o_t, h_t

    def rnn_layer_forward(self, Xt):

        batch, input_dim_size, T_x = Xt.shape
        # allocate H, and O for the first time
        if (self.H is None and self.O is None):
            print('first time call!')
            self.H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
            self.O = np.zeros(shape=(batch, self.outputsize, T_x))

        if (self.mode == 1):
            h_previous = np.zeros(shape=(batch, self.hidden_state_size))
        else:
            h_previous = self.H[:, :, -1]

        for t in range(T_x):
            output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous) 
            self.H[:, :, t] = h_t
            self.O[:, :, t] = output

            # Our current/new hiddenstate will be the previous hiddenstate for the next round,
            h_previous = h_t

        return self.O, self.H

    def rnn_cell_backward(self, xt, h, output, true_label, h_gradient_or_dh, t):
        """
        h: is the next state 
        output: is the output of the current cell!
        true_label: is the true label for the current output
        h_gradient_or_dh: is the dh which in the beginning is zero
                         and is updated as we go backward in the backprogagation
                         remember the backward pass is essentially a loop! 
                         so have that in mind and you will be fine!
        """

        e = output
        e[0,t] = output[0,t] - true_label[0,t]
        per_ts_loss = output[0,t] - true_label[0,t]

        # must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
        dW3 = np.dot(e.T, h)
        dh = np.dot(e, self.W3) + h_gradient_or_dh  # from later cell
        # dbo +=e.1, we use sum, becasue we have a batch
        #e is a vector, when it is subtracted from label, the result will be added to dbo
        dbo = np.sum(e, axis=0)# e.reshape(e.shape[1])#np.sum(e, axis=0)
        # print('e=',e)

        # the input part
        dtanh = (1 - h**2) * dh

        # compute the gradient of the loss with respect to W1
        dxt = np.dot(dtanh, self.W1.T)
        # must have the shape of (vocab_size, hidden_state_size)
        dW1 = np.dot(xt.T, dtanh)

        # compute the gradient with respect to W2
        dh_prev = np.dot(dtanh, self.W2)
        # shape must be (HiddenSize, HiddenSize)
        dW2 = np.dot(dh_prev.T, dtanh)

        #print('dtanh.shape = ',dtanh.shape)
        # dbh += dtanh.1, we use sum, since we have a batch
        dbh = np.sum(dtanh, axis=0)

        return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss

    def rnn_layer_backward(self, Xt, labels, H, O):

        dW1 = np.zeros_like(self.W1)
        dW2 = np.zeros_like(self.W2)
        dW3 = np.zeros_like(self.W3)
        # must have the shape(batch,hiddensize)
        dBh = np.zeros_like(self.bh)
        # must have the shape(batch,outputsize)
        dBo = np.zeros_like(self.bo)
        dH_prev = np.zeros_like(H[:, :, 0])
        dh = np.zeros_like(H[:, :, 0])
        dXt = np.zeros_like(Xt)

        _, _, T_x = Xt.shape
        loss = 0

        for t in reversed(range(T_x)):
            dw1, dw2, dw3, dbh, dbo, dxt, dh_prev, dh, e = self.rnn_cell_backward(
                                                                                Xt[:, :, t],
                                                                                 H[:, :, t],
                                                                                 O[:, :, t],
                                                                            labels[:, :, t],
                                                                            dh, t)
            dW1 += dw1
            dW2 += dw2
            dW3 += dw3
            dBh += dbh
            dBo += dbo
            dH_prev += dH_prev
            # Update the loss by substracting the cross-entropy term of this time-step from it.
            loss -= np.log(e)#e.sum())

        return dW1, dW2, dW3, dbh, dBo, dXt, dH_prev, dh, loss

    def update_parameters(self, dW1, dW2, dW3, dbh, dbo, lr):
        """
        updates the parameters 
        lr : 
        gradients
        """

        self.W1 += -lr * dW1
        self.W2 += -lr * dW2
        self.W3 += -lr * dW3
        self.bh += -lr * dbh
        self.bo += -lr * dbo

        return self.W1, self.W2, self.W3, self.bh, self.bo

    # update
    def optimize(self, X, Y, learning_rate=0.01):

        # Forward propagate through time 
        O, H = self.rnn_layer_forward(X)  

        # Backpropagate through time 
        dW1, dW2, dW3, dbh, dbo, dXt, dH_prev, dh, loss = self.rnn_layer_backward(
            X, Y, H, O) 

        # Clip your gradients between -5 (min) and 5 (max) 
        dW1, dW2, dW3, dbh, dbo = clip(dW1, dW2, dW3, dbh, dbo, maxValue=5)

        # Update parameters
        W1, dW2, dW3, dbh, dbo = self.update_parameters(
            dW1, dW2, dW3, dbh, dbo, learning_rate)

        return loss, dW1, dW2, dW3, dbh, dbo

in case someone needs a quick reproducible code, this is the whole jupyter notebook(ready example) : https://paste.ee/p/iOnbk
and this is the dinos.txt file : https://paste.ee/p/AIdSH

Update 1:

Thanks to God! I found two issues in the code that were responsible for the weird error message! i.e all outputs being 1, which would later result in a messed up back-propagation.
The actual problem was caused by the softmax function I had up there! The one that I was using was from Andrew Ng's code.
In his code, he only uses a batch of 1 and his code wont work with higher number of batches it seems. I was using batches so I had to fix that. instead I had to use scikit learn's softmax to fix the issue:

from sklearn.utils.extmath import softmax

or the snippet below (taken from this gentelman) :

def softmax(x):
    e_x = np.exp(x.T - np.max(x, axis = -1))
    return (e_x / e_x.sum(axis=0)).T

The second problem of mine, was actually the wrong labels. I had mistakenly wrote :

X = X[index:index+batchsize,:,:]  
Y = np.zeros_like(X)
Y[index:index+batchsize,:-1,:] = X[index:index+batchsize,1:,:]

Whereas I must have been doing it like :

X = X[index:index+batchsize,:,:]  
Y = np.zeros_like(X)
Y[index:index+batchsize,:,:-1] = X[index:index+batchsize,:,1:]

These were the issues responsible for the error specified above. However the loss now, increases instead of decreasing! and I dont know which route to take. and if this increase can be attributed to the wrong back-propagation?

Any help is greatly appreciated.

$\endgroup$
0
$\begingroup$

Thanks to God! I finally found the issue and here it is . First of all, since this is a sequence generation example, and all samples do not have the same length, we cant use batches of samples. so we use batch of 1.
Technically speaking we can use batches, but there is a catch!
One solution is to use the biggest length for all sequences.
This will only work in two cases :

  1. Your sequence is very short when you are dealing with vanilla rnn.
  2. If you have long sequences, you must use LSTM or GRU units.

Since we are using the vanilla rnn, and our sequences are relatively long (between 11 to 27 characters long), we cant use the same length for everyone.
This means that in the back-propagation part, we wont be subtracting the whole output from the corresponding label, rather we only check the index of the correct word in our output against our label(np.argmax() of our label) and see if at that index, our output was correct or not(or simply subtracting that index in our output from 1)

output[0,np.argmax(label)] -= 1 

or

output[0,np.argmax(label)] -= label[0,np.argmax(label)]

I also made a mistake in backpropagation section here : instead of using the h_previous, I used dh_previous. so the correct form was :

def rnn_cell_backward(self, xt, h, h_prev, output, true_label, h_gradient_or_dh):
    """
    h: is the next state 
    output: is the output of the current cell!
    true_label: is the true label for the current output
    h_gradient_or_dh: is the dh which in the beginning is zero
                     and is updated as we go backward in the backprogagation
                     remember the backward pass is essentially a loop! 
                     so have that in mind and you will be fine!
    """

    e = np.copy(output)
    e[0,np.argmax(true_label)] -= true_label[0,np.argmax(true_label)]
    per_ts_loss = e[0,np.argmax(true_label)]

    # must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
    dW3 = np.dot(e.T, h)
    dh = np.dot(e, self.W3) + h_gradient_or_dh  # from later cell
    # dbo +=e.1, we use sum, because we have a batch
    #e is a vector, when it is subtracted from label, the result will be added to dbo
    # since batchsize is always 1 we dont need to do np.sum(e, axis=0)
    dbo = e.reshape(e.shape[1])

    # the input part
    dtanh = (1 - h**2) * dh

    # compute the gradient of the loss with respect to W1
    dxt = np.dot(dtanh, self.W1.T)
    # must have the shape of (vocab_size, hidden_state_size)
    dW1 = np.dot(xt.T, dtanh)

    # compute the gradient with respect to W2
    dh_prev = np.dot(dtanh, self.W2)
    # shape must be (HiddenSize, HiddenSize)
    dW2 = np.dot(h_prev.T, dtanh)

    #print('dtanh.shape = ',dtanh.shape)
    # dbh += dtanh.1, we use sum, since we have a batch
    dbh = np.sum(dtanh, axis=0)

    return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss

There was also something wrong in the forward pass. Since the sequence length is not the same, we cant have a prespecified H, and O. and thus making them a class field serves nor purpose. Also, we can have 2 options when feeding the h_previous to the calculations.

  1. use zero initialized h_previous always
  2. use the last hidden state of calculated by previous sample.

Since our examples are not related in anyway, we should use the first option. My first impression was that if we go the second route, we will see the network cant perform decently at all and fails to create meaningful outputs, whereas in the case of the first method, it can quickly learn to produce good outputs. I was wrong!! at least in this example, they create the same exact outputs!
So the rnn_layer_forward should look like this instead :

def rnn_layer_forward(self, Xt, H=None):

    batch, input_dim_size, T_x = Xt.shape
    # allocate H, and O for the first time
    H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
    O = np.zeros(shape=(batch, self.outputsize, T_x))

    if (self.mode == 1):
        h_previous = np.zeros(shape=(batch, self.hidden_state_size))
    else:
        h_previous = self.H[:, :, -1]

    for t in range(T_x):
        output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous) 
        self.H[:, :, t] = h_t
        self.O[:, :, t] = output

        # Our current/new hiddenstate will be the previous hiddenstate for the next round,
        h_previous = h_t

    return O, H
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.