I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself.
However, it seems I have not understood some parts as it was not covered or explained in the course. Right now, I don't know if my implementation is wrong or some it is some hyper parameter tuning issue since when trying a simple example, I get weird error in which very soon, all entries reach 1 (as the confidence of a specific class!). I explain shortly.
But before I get to that, I want to reiterate my understanding of how backward phase is carried out in a simple RNN layer.
In forward pass we simply calculate two terms:
$h_t = tanh(W_1.Xt + W_2.h_{t-1} + b_h)$
$output_t = softmax(W3.h_t + b_o)$
Although in practice the dimensions are different and instead the terms look like this :
$h_t = tanh(Xt.W_1 + h_{t-1}.W_2 + b_h)$
$output_t = softmax(h_t.W3.T + b_o)$
This is the python implementation that I wrote :
def rnn_cell_foward(self, xt, h0):
"""
Run the forward pass for a single timestep of a vanilla RNN that uses a tanh
activation function.
The input data has dimension D(vocabsize in case we have nlp use), the hidden state has dimension H, and we use
a minibatch size of N.
Inputs:
- x: Input data for this timestep, of shape (Batch_size, vocabsize_or_basically_input_dim_size).
- h0: Hidden state from previous timestep, of shape (Batch_size, HiddenSize)
- W1: Weight matrix for input-to-hidden connections, of shape (vocabsize, HiddenSize)
- W2: Weight matrix for hidden-to-hidden connections, of shape (HiddenSize, HiddenSize)
- W3: Weight matrix for hiddent-to-output connections, of shape(vocabsize_or_output_dim_size, hidden_state_size)
- bh: Biases of shape (HiddenSize,)
- bo: Biases of shape (vocabsize_or_output_dim_size,)
Returns a tuple of:
- next_h: Next hidden state, of shape (Bachsize, HiddenSize)
- output: output
"""
h_t = np.tanh(np.dot(xt, self.W1) + np.dot(h0, self.W2) + self.bh)
o_t = softmax(np.dot(h_t, self.W3.T) + self.bo)
return o_t, h_t
def rnn_layer_forward(self, Xt):
batch, input_dim_size, T_x = Xt.shape
# allocate H, and O for the first time
if (self.H is None and self.O is None):
print('first time call!')
self.H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
self.O = np.zeros(shape=(batch, self.outputsize, T_x))
if (self.mode == 1):
h_previous = np.zeros(shape=(batch, self.hidden_state_size))
else:
h_previous = self.H[:, :, -1]
for t in range(T_x):
output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous)
self.H[:, :, t] = h_t
self.O[:, :, t] = output
# Our current/new hiddenstate will be the previous hiddenstate for the next round,
h_previous = h_t
return self.O, self.H
So far so good. Now for the Back-propagation, we need to do the following :
Calculate the error by subtracting the output from the label (
e = output - label)Calculate the gradients for the output term
- Calculate the gradients for the hidden state term.
- For this we start from the last time-step and loop back to the start.
Now this is where my confusion begins. first of all, how am I supposed to subtract the output? do I simple do :
e = output - label
or should I specifically subtract the entry equal to the current time step? i.e :
e = output[t] - label[t]
(especially when we are dealing with sequence to sequence rnn, when we have a sentence/word as input, and we want to generate a sentence/word as the output)
If I go the first route, $e$ will be a vector of size $output$. and for loss, I need to sum all of the entries. Is this expected behavior?
If I go the second route, $e$ will be a single number, I can use it for the loss, but is it the right way to do so?
Since if we go the second route, very quickly our network would learn just to set each entry to 1 regardless of other entries and that is not good. in fact this is what I'm getting now I guess.
I have seen this implementation in Andrew Ng's implementation and that's why its confusing me! for him it works at least, but mine it is not! and also it doesn't make any sense.
When we use a batch of samples instead of 1, what should I do then? in the first route, I'd just sum everything, do I need to sum all errors here as well?
This error is important because it will be multiplied by the gradients of the output term (i.e $dW_3, dh_t, db_o$)
I tried both scenarios and both failed. Below is the methods I wrote for Back-propagation :
def rnn_cell_backward(self, xt, h, output, true_label, h_gradient_or_dh, t):
"""
h: is the next state
output: is the output of the current cell!
true_label: is the true label for the current output
h_gradient_or_dh: is the dh which in the beginning is zero
and is updated as we go backward in the backprogagation
remember the backward pass is essentially a loop!
so have that in mind and you will be fine!
"""
e = output
e[0,t] = output[0,t] - true_label[0,t]
per_ts_loss = output[0,t] - true_label[0,t]
# must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
dW3 = np.dot(e.T, h)
dh = np.dot(e, self.W3) + h_gradient_or_dh # from later cell
# dbo +=e.1, we use sum, becasue we have a batch
#e is a vector, when it is subtracted from label, the result will be added to dbo
dbo = np.sum(e, axis=0) #e.reshape(e.shape[1]) #np.sum(e, axis=0)
# print('e=',e)
# the input part
dtanh = (1 - h**2) * dh
# compute the gradient of the loss with respect to W1
dxt = np.dot(dtanh, self.W1.T)
# must have the shape of (vocab_size, hidden_state_size)
dW1 = np.dot(xt.T, dtanh)
# compute the gradient with respect to W2
dh_prev = np.dot(dtanh, self.W2)
# shape must be (HiddenSize, HiddenSize)
dW2 = np.dot(dh_prev.T, dtanh)
#print('dtanh.shape = ',dtanh.shape)
# dbh += dtanh.1, we use sum, since we have a batch
dbh = np.sum(dtanh, axis=0)
return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss
def rnn_layer_backward(self, Xt, labels, H, O):
dW1 = np.zeros_like(self.W1)
dW2 = np.zeros_like(self.W2)
dW3 = np.zeros_like(self.W3)
# must have the shape(batch,hiddensize)
dBh = np.zeros_like(self.bh)
# must have the shape(batch,outputsize)
dBo = np.zeros_like(self.bo)
dH_prev = np.zeros_like(H[:, :, 0])
dh = np.zeros_like(H[:, :, 0])
dXt = np.zeros_like(Xt)
_, _, T_x = Xt.shape
loss = 0
for t in reversed(range(T_x)):
dw1, dw2, dw3, dbh, dbo, dxt, dh_prev, dh, e = self.rnn_cell_backward(
Xt[:, :, t],
H[:, :, t],
O[:, :, t],
labels[:, :, t],
dh, t)
dW1 += dw1
dW2 += dw2
dW3 += dw3
dBh += dbh
dBo += dbo
dH_prev += dH_prev
# Update the loss by subtracting the cross-entropy term of this time-step from it.
loss -= np.log(e)#e.sum())
return dW1, dW2, dW3, dbh, dBo, dXt, dH_prev, dh, loss
Now for the test, I tried the Andrew Ng's dinosaur name generation example. but at the very beginning I get this weird error where all outputs are 1!
Any help is greatly appreciated.
By the way this is my simple rnn definition:
import numpy as np
def softmax(x):
e_x = np.exp(x - np.max(x))
return e_x / e_x.sum(axis=0)
class RNNClass(object):
def __init__(self, vocab_size, outputsize, hidden_state_size=100, mode=1):
np.random.seed(1)
self.W1 = np.random.randn(vocab_size, hidden_state_size) * 0.01
self.W2 = np.random.randn(hidden_state_size, hidden_state_size) * 0.01
self.W3 = np.random.randn(outputsize, hidden_state_size) * 0.01
self.bh = np.random.randn(hidden_state_size)
self.bo = np.random.randn(outputsize)
self.outputsize = outputsize
self.hidden_state_size = hidden_state_size
self.H = None
self.O = None
self.mode = mode
print('W1: ', self.W1.shape)
print('W2: ', self.W2.shape)
print('W3: ', self.W3.shape)
print('bh: ', self.bh.shape)
print('bo: ', self.bo.shape)
import numpy as np
def rnn_cell_foward(self, xt, h0):
"""
Run the forward pass for a single timestep of a vanilla RNN that uses a tanh
activation function.
The input data has dimension D(vocabsize in case we have nlp use), the hidden state has dimension H, and we use
a minibatch size of N.
Inputs:
- x: Input data for this timestep, of shape (Batch_size, vocabsize_or_basically_input_dim_size).
- h0: Hidden state from previous timestep, of shape (Batch_size, HiddenSize)
- W1: Weight matrix for input-to-hidden connections, of shape (vocabsize, HiddenSize)
- W2: Weight matrix for hidden-to-hidden connections, of shape (HiddenSize, HiddenSize)
- W3: Weight matrix for hiddent-to-output connections, of shape(vocabsize_or_output_dim_size, hidden_state_size)
- bh: Biases of shape (HiddenSize,)
- bo: Biases of shape (vocabsize_or_output_dim_size,)
Returns a tuple of:
- next_h: Next hidden state, of shape (Bachsize, HiddenSize)
- output: output
"""
h_t = np.tanh(np.dot(xt, self.W1) + np.dot(h0, self.W2) + self.bh)
o_t = softmax(np.dot(h_t, self.W3.T) + self.bo)
return o_t, h_t
def rnn_layer_forward(self, Xt):
batch, input_dim_size, T_x = Xt.shape
# allocate H, and O for the first time
if (self.H is None and self.O is None):
print('first time call!')
self.H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
self.O = np.zeros(shape=(batch, self.outputsize, T_x))
if (self.mode == 1):
h_previous = np.zeros(shape=(batch, self.hidden_state_size))
else:
h_previous = self.H[:, :, -1]
for t in range(T_x):
output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous)
self.H[:, :, t] = h_t
self.O[:, :, t] = output
# Our current/new hiddenstate will be the previous hiddenstate for the next round,
h_previous = h_t
return self.O, self.H
def rnn_cell_backward(self, xt, h, output, true_label, h_gradient_or_dh, t):
"""
h: is the next state
output: is the output of the current cell!
true_label: is the true label for the current output
h_gradient_or_dh: is the dh which in the beginning is zero
and is updated as we go backward in the backprogagation
remember the backward pass is essentially a loop!
so have that in mind and you will be fine!
"""
e = output
e[0,t] = output[0,t] - true_label[0,t]
per_ts_loss = output[0,t] - true_label[0,t]
# must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
dW3 = np.dot(e.T, h)
dh = np.dot(e, self.W3) + h_gradient_or_dh # from later cell
# dbo +=e.1, we use sum, becasue we have a batch
#e is a vector, when it is subtracted from label, the result will be added to dbo
dbo = np.sum(e, axis=0)# e.reshape(e.shape[1])#np.sum(e, axis=0)
# print('e=',e)
# the input part
dtanh = (1 - h**2) * dh
# compute the gradient of the loss with respect to W1
dxt = np.dot(dtanh, self.W1.T)
# must have the shape of (vocab_size, hidden_state_size)
dW1 = np.dot(xt.T, dtanh)
# compute the gradient with respect to W2
dh_prev = np.dot(dtanh, self.W2)
# shape must be (HiddenSize, HiddenSize)
dW2 = np.dot(dh_prev.T, dtanh)
#print('dtanh.shape = ',dtanh.shape)
# dbh += dtanh.1, we use sum, since we have a batch
dbh = np.sum(dtanh, axis=0)
return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss
def rnn_layer_backward(self, Xt, labels, H, O):
dW1 = np.zeros_like(self.W1)
dW2 = np.zeros_like(self.W2)
dW3 = np.zeros_like(self.W3)
# must have the shape(batch,hiddensize)
dBh = np.zeros_like(self.bh)
# must have the shape(batch,outputsize)
dBo = np.zeros_like(self.bo)
dH_prev = np.zeros_like(H[:, :, 0])
dh = np.zeros_like(H[:, :, 0])
dXt = np.zeros_like(Xt)
_, _, T_x = Xt.shape
loss = 0
for t in reversed(range(T_x)):
dw1, dw2, dw3, dbh, dbo, dxt, dh_prev, dh, e = self.rnn_cell_backward(
Xt[:, :, t],
H[:, :, t],
O[:, :, t],
labels[:, :, t],
dh, t)
dW1 += dw1
dW2 += dw2
dW3 += dw3
dBh += dbh
dBo += dbo
dH_prev += dH_prev
# Update the loss by substracting the cross-entropy term of this time-step from it.
loss -= np.log(e)#e.sum())
return dW1, dW2, dW3, dbh, dBo, dXt, dH_prev, dh, loss
def update_parameters(self, dW1, dW2, dW3, dbh, dbo, lr):
"""
updates the parameters
lr :
gradients
"""
self.W1 += -lr * dW1
self.W2 += -lr * dW2
self.W3 += -lr * dW3
self.bh += -lr * dbh
self.bo += -lr * dbo
return self.W1, self.W2, self.W3, self.bh, self.bo
# update
def optimize(self, X, Y, learning_rate=0.01):
# Forward propagate through time
O, H = self.rnn_layer_forward(X)
# Backpropagate through time
dW1, dW2, dW3, dbh, dbo, dXt, dH_prev, dh, loss = self.rnn_layer_backward(
X, Y, H, O)
# Clip your gradients between -5 (min) and 5 (max)
dW1, dW2, dW3, dbh, dbo = clip(dW1, dW2, dW3, dbh, dbo, maxValue=5)
# Update parameters
W1, dW2, dW3, dbh, dbo = self.update_parameters(
dW1, dW2, dW3, dbh, dbo, learning_rate)
return loss, dW1, dW2, dW3, dbh, dbo
in case someone needs a quick reproducible code, this is the whole jupyter notebook(ready example) : https://paste.ee/p/iOnbk
and this is the dinos.txt file : https://paste.ee/p/AIdSH
Update 1:
Thanks to God! I found two issues in the code that were responsible for the weird error message! i.e all outputs being 1, which would later result in a messed up back-propagation.
The actual problem was caused by the softmax function I had up there! The one that I was using was from Andrew Ng's code.
In his code, he only uses a batch of 1 and his code wont work with higher number of batches it seems. I was using batches so I had to fix that. instead I had to use scikit learn's softmax to fix the issue:
from sklearn.utils.extmath import softmax
or the snippet below (taken from this gentelman) :
def softmax(x):
e_x = np.exp(x.T - np.max(x, axis = -1))
return (e_x / e_x.sum(axis=0)).T
The second problem of mine, was actually the wrong labels. I had mistakenly wrote :
X = X[index:index+batchsize,:,:]
Y = np.zeros_like(X)
Y[index:index+batchsize,:-1,:] = X[index:index+batchsize,1:,:]
Whereas I must have been doing it like :
X = X[index:index+batchsize,:,:]
Y = np.zeros_like(X)
Y[index:index+batchsize,:,:-1] = X[index:index+batchsize,:,1:]
These were the issues responsible for the error specified above. However the loss now, increases instead of decreasing! and I dont know which route to take. and if this increase can be attributed to the wrong back-propagation?
Any help is greatly appreciated.
