# How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself.
However, it seems I have not understood some parts as it was not covered or explained in the course. Right now, I don't know if my implementation is wrong or some it is some hyper parameter tuning issue since when trying a simple example, I get weird error in which very soon, all entries reach 1 (as the confidence of a specific class!). I explain shortly.
But before I get to that, I want to reiterate my understanding of how backward phase is carried out in a simple RNN layer.

In forward pass we simply calculate two terms:

$$h_t = tanh(W_1.Xt + W_2.h_{t-1} + b_h)$$
$$output_t = softmax(W3.h_t + b_o)$$

Although in practice the dimensions are different and instead the terms look like this :

$$h_t = tanh(Xt.W_1 + h_{t-1}.W_2 + b_h)$$
$$output_t = softmax(h_t.W3.T + b_o)$$

This is the python implementation that I wrote :

def rnn_cell_foward(self, xt, h0):
"""
Run the forward pass for a single timestep of a vanilla RNN that uses a tanh
activation function.
The input data has dimension D(vocabsize in case we have nlp use), the hidden state has dimension H, and we use
a minibatch size of N.
Inputs:
- x: Input data for this timestep, of shape (Batch_size, vocabsize_or_basically_input_dim_size).
- h0: Hidden state from previous timestep, of shape (Batch_size, HiddenSize)
- W1: Weight matrix for input-to-hidden connections, of shape (vocabsize, HiddenSize)
- W2: Weight matrix for hidden-to-hidden connections, of shape (HiddenSize, HiddenSize)
- W3: Weight matrix for hiddent-to-output connections, of shape(vocabsize_or_output_dim_size, hidden_state_size)
- bh: Biases of shape (HiddenSize,)
- bo: Biases of shape (vocabsize_or_output_dim_size,)

Returns a tuple of:
- next_h: Next hidden state, of shape (Bachsize, HiddenSize)
- output: output
"""

h_t = np.tanh(np.dot(xt, self.W1) + np.dot(h0, self.W2) + self.bh)
o_t = softmax(np.dot(h_t, self.W3.T) + self.bo)
return o_t, h_t

def rnn_layer_forward(self, Xt):

batch, input_dim_size, T_x = Xt.shape
# allocate H, and O for the first time
if (self.H is None and self.O is None):
print('first time call!')
self.H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
self.O = np.zeros(shape=(batch, self.outputsize, T_x))

if (self.mode == 1):
h_previous = np.zeros(shape=(batch, self.hidden_state_size))
else:
h_previous = self.H[:, :, -1]

for t in range(T_x):
output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous)
self.H[:, :, t] = h_t
self.O[:, :, t] = output

# Our current/new hiddenstate will be the previous hiddenstate for the next round,
h_previous = h_t

return self.O, self.H


So far so good. Now for the Back-propagation, we need to do the following :

1. Calculate the error by subtracting the output from the label (e = output - label)

2. Calculate the gradients for the output term

3. Calculate the gradients for the hidden state term.
4. For this we start from the last time-step and loop back to the start.

Now this is where my confusion begins. first of all, how am I supposed to subtract the output? do I simple do :

e = output - label


or should I specifically subtract the entry equal to the current time step? i.e :

e = output[t] - label[t]


(especially when we are dealing with sequence to sequence rnn, when we have a sentence/word as input, and we want to generate a sentence/word as the output)

If I go the first route, $$e$$ will be a vector of size $$output$$. and for loss, I need to sum all of the entries. Is this expected behavior?
If I go the second route, $$e$$ will be a single number, I can use it for the loss, but is it the right way to do so? Since if we go the second route, very quickly our network would learn just to set each entry to 1 regardless of other entries and that is not good. in fact this is what I'm getting now I guess. I have seen this implementation in Andrew Ng's implementation and that's why its confusing me! for him it works at least, but mine it is not! and also it doesn't make any sense.
When we use a batch of samples instead of 1, what should I do then? in the first route, I'd just sum everything, do I need to sum all errors here as well?

This error is important because it will be multiplied by the gradients of the output term (i.e $$dW_3, dh_t, db_o$$)

I tried both scenarios and both failed. Below is the methods I wrote for Back-propagation :

 def rnn_cell_backward(self, xt, h, output, true_label, h_gradient_or_dh, t):
"""
h: is the next state
output: is the output of the current cell!
true_label: is the true label for the current output
h_gradient_or_dh: is the dh which in the beginning is zero
and is updated as we go backward in the backprogagation
remember the backward pass is essentially a loop!
so have that in mind and you will be fine!
"""

e = output
e[0,t] = output[0,t] - true_label[0,t]
per_ts_loss = output[0,t] - true_label[0,t]

# must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
dW3 = np.dot(e.T, h)
dh = np.dot(e, self.W3) + h_gradient_or_dh  # from later cell
# dbo +=e.1, we use sum, becasue we have a batch
#e is a vector, when it is subtracted from label, the result will be added to dbo
dbo = np.sum(e, axis=0) #e.reshape(e.shape) #np.sum(e, axis=0)
# print('e=',e)

# the input part
dtanh = (1 - h**2) * dh

# compute the gradient of the loss with respect to W1
dxt = np.dot(dtanh, self.W1.T)
# must have the shape of (vocab_size, hidden_state_size)
dW1 = np.dot(xt.T, dtanh)

# compute the gradient with respect to W2
dh_prev = np.dot(dtanh, self.W2)
# shape must be (HiddenSize, HiddenSize)
dW2 = np.dot(dh_prev.T, dtanh)

#print('dtanh.shape = ',dtanh.shape)
# dbh += dtanh.1, we use sum, since we have a batch
dbh = np.sum(dtanh, axis=0)

return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss

def rnn_layer_backward(self, Xt, labels, H, O):

dW1 = np.zeros_like(self.W1)
dW2 = np.zeros_like(self.W2)
dW3 = np.zeros_like(self.W3)
# must have the shape(batch,hiddensize)
dBh = np.zeros_like(self.bh)
# must have the shape(batch,outputsize)
dBo = np.zeros_like(self.bo)
dH_prev = np.zeros_like(H[:, :, 0])
dh = np.zeros_like(H[:, :, 0])
dXt = np.zeros_like(Xt)

_, _, T_x = Xt.shape
loss = 0

for t in reversed(range(T_x)):
dw1, dw2, dw3, dbh, dbo, dxt, dh_prev, dh, e = self.rnn_cell_backward(
Xt[:, :, t],
H[:, :, t],
O[:, :, t],
labels[:, :, t],
dh, t)
dW1 += dw1
dW2 += dw2
dW3 += dw3
dBh += dbh
dBo += dbo
dH_prev += dH_prev
# Update the loss by subtracting the cross-entropy term of this time-step from it.
loss -= np.log(e)#e.sum())

return dW1, dW2, dW3, dbh, dBo, dXt, dH_prev, dh, loss


Now for the test, I tried the Andrew Ng's dinosaur name generation example. but at the very beginning I get this weird error where all outputs are 1!

Any help is greatly appreciated.
By the way this is my simple rnn definition:

import numpy as np

def softmax(x):
e_x = np.exp(x - np.max(x))
return e_x / e_x.sum(axis=0)

class RNNClass(object):
def __init__(self, vocab_size, outputsize, hidden_state_size=100, mode=1):

np.random.seed(1)
self.W1 = np.random.randn(vocab_size, hidden_state_size) * 0.01
self.W2 = np.random.randn(hidden_state_size, hidden_state_size) * 0.01
self.W3 = np.random.randn(outputsize, hidden_state_size) * 0.01
self.bh = np.random.randn(hidden_state_size)
self.bo = np.random.randn(outputsize)

self.outputsize = outputsize
self.hidden_state_size = hidden_state_size
self.H = None
self.O = None
self.mode = mode

print('W1: ', self.W1.shape)
print('W2: ', self.W2.shape)
print('W3: ', self.W3.shape)
print('bh: ', self.bh.shape)
print('bo: ', self.bo.shape)

import numpy as np

def rnn_cell_foward(self, xt, h0):
"""
Run the forward pass for a single timestep of a vanilla RNN that uses a tanh
activation function.
The input data has dimension D(vocabsize in case we have nlp use), the hidden state has dimension H, and we use
a minibatch size of N.
Inputs:
- x: Input data for this timestep, of shape (Batch_size, vocabsize_or_basically_input_dim_size).
- h0: Hidden state from previous timestep, of shape (Batch_size, HiddenSize)
- W1: Weight matrix for input-to-hidden connections, of shape (vocabsize, HiddenSize)
- W2: Weight matrix for hidden-to-hidden connections, of shape (HiddenSize, HiddenSize)
- W3: Weight matrix for hiddent-to-output connections, of shape(vocabsize_or_output_dim_size, hidden_state_size)
- bh: Biases of shape (HiddenSize,)
- bo: Biases of shape (vocabsize_or_output_dim_size,)

Returns a tuple of:
- next_h: Next hidden state, of shape (Bachsize, HiddenSize)
- output: output
"""

h_t = np.tanh(np.dot(xt, self.W1) + np.dot(h0, self.W2) + self.bh)
o_t = softmax(np.dot(h_t, self.W3.T) + self.bo)
return o_t, h_t

def rnn_layer_forward(self, Xt):

batch, input_dim_size, T_x = Xt.shape
# allocate H, and O for the first time
if (self.H is None and self.O is None):
print('first time call!')
self.H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
self.O = np.zeros(shape=(batch, self.outputsize, T_x))

if (self.mode == 1):
h_previous = np.zeros(shape=(batch, self.hidden_state_size))
else:
h_previous = self.H[:, :, -1]

for t in range(T_x):
output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous)
self.H[:, :, t] = h_t
self.O[:, :, t] = output

# Our current/new hiddenstate will be the previous hiddenstate for the next round,
h_previous = h_t

return self.O, self.H

def rnn_cell_backward(self, xt, h, output, true_label, h_gradient_or_dh, t):
"""
h: is the next state
output: is the output of the current cell!
true_label: is the true label for the current output
h_gradient_or_dh: is the dh which in the beginning is zero
and is updated as we go backward in the backprogagation
remember the backward pass is essentially a loop!
so have that in mind and you will be fine!
"""

e = output
e[0,t] = output[0,t] - true_label[0,t]
per_ts_loss = output[0,t] - true_label[0,t]

# must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
dW3 = np.dot(e.T, h)
dh = np.dot(e, self.W3) + h_gradient_or_dh  # from later cell
# dbo +=e.1, we use sum, becasue we have a batch
#e is a vector, when it is subtracted from label, the result will be added to dbo
dbo = np.sum(e, axis=0)# e.reshape(e.shape)#np.sum(e, axis=0)
# print('e=',e)

# the input part
dtanh = (1 - h**2) * dh

# compute the gradient of the loss with respect to W1
dxt = np.dot(dtanh, self.W1.T)
# must have the shape of (vocab_size, hidden_state_size)
dW1 = np.dot(xt.T, dtanh)

# compute the gradient with respect to W2
dh_prev = np.dot(dtanh, self.W2)
# shape must be (HiddenSize, HiddenSize)
dW2 = np.dot(dh_prev.T, dtanh)

#print('dtanh.shape = ',dtanh.shape)
# dbh += dtanh.1, we use sum, since we have a batch
dbh = np.sum(dtanh, axis=0)

return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss

def rnn_layer_backward(self, Xt, labels, H, O):

dW1 = np.zeros_like(self.W1)
dW2 = np.zeros_like(self.W2)
dW3 = np.zeros_like(self.W3)
# must have the shape(batch,hiddensize)
dBh = np.zeros_like(self.bh)
# must have the shape(batch,outputsize)
dBo = np.zeros_like(self.bo)
dH_prev = np.zeros_like(H[:, :, 0])
dh = np.zeros_like(H[:, :, 0])
dXt = np.zeros_like(Xt)

_, _, T_x = Xt.shape
loss = 0

for t in reversed(range(T_x)):
dw1, dw2, dw3, dbh, dbo, dxt, dh_prev, dh, e = self.rnn_cell_backward(
Xt[:, :, t],
H[:, :, t],
O[:, :, t],
labels[:, :, t],
dh, t)
dW1 += dw1
dW2 += dw2
dW3 += dw3
dBh += dbh
dBo += dbo
dH_prev += dH_prev
# Update the loss by substracting the cross-entropy term of this time-step from it.
loss -= np.log(e)#e.sum())

return dW1, dW2, dW3, dbh, dBo, dXt, dH_prev, dh, loss

def update_parameters(self, dW1, dW2, dW3, dbh, dbo, lr):
"""
lr :
"""

self.W1 += -lr * dW1
self.W2 += -lr * dW2
self.W3 += -lr * dW3
self.bh += -lr * dbh
self.bo += -lr * dbo

return self.W1, self.W2, self.W3, self.bh, self.bo

# update
def optimize(self, X, Y, learning_rate=0.01):

# Forward propagate through time
O, H = self.rnn_layer_forward(X)

# Backpropagate through time
dW1, dW2, dW3, dbh, dbo, dXt, dH_prev, dh, loss = self.rnn_layer_backward(
X, Y, H, O)

dW1, dW2, dW3, dbh, dbo = clip(dW1, dW2, dW3, dbh, dbo, maxValue=5)

# Update parameters
W1, dW2, dW3, dbh, dbo = self.update_parameters(
dW1, dW2, dW3, dbh, dbo, learning_rate)

return loss, dW1, dW2, dW3, dbh, dbo


in case someone needs a quick reproducible code, this is the whole jupyter notebook(ready example) : https://paste.ee/p/iOnbk
and this is the dinos.txt file : https://paste.ee/p/AIdSH

Update 1:

Thanks to God! I found two issues in the code that were responsible for the weird error message! i.e all outputs being 1, which would later result in a messed up back-propagation.
The actual problem was caused by the softmax function I had up there! The one that I was using was from Andrew Ng's code.
In his code, he only uses a batch of 1 and his code wont work with higher number of batches it seems. I was using batches so I had to fix that. instead I had to use scikit learn's softmax to fix the issue:

from sklearn.utils.extmath import softmax


or the snippet below (taken from this gentelman) :

def softmax(x):
e_x = np.exp(x.T - np.max(x, axis = -1))
return (e_x / e_x.sum(axis=0)).T


The second problem of mine, was actually the wrong labels. I had mistakenly wrote :

X = X[index:index+batchsize,:,:]
Y = np.zeros_like(X)
Y[index:index+batchsize,:-1,:] = X[index:index+batchsize,1:,:]


Whereas I must have been doing it like :

X = X[index:index+batchsize,:,:]
Y = np.zeros_like(X)
Y[index:index+batchsize,:,:-1] = X[index:index+batchsize,:,1:]


These were the issues responsible for the error specified above. However the loss now, increases instead of decreasing! and I dont know which route to take. and if this increase can be attributed to the wrong back-propagation?

Any help is greatly appreciated.

Thanks to God! I finally found the issue and here it is . First of all, since this is a sequence generation example, and all samples do not have the same length, we cant use batches of samples. so we use batch of 1.
Technically speaking we can use batches, but there is a catch!
One solution is to use the biggest length for all sequences.
This will only work in two cases :

1. Your sequence is very short when you are dealing with vanilla rnn.
2. If you have long sequences, you must use LSTM or GRU units.

Since we are using the vanilla rnn, and our sequences are relatively long (between 11 to 27 characters long), we cant use the same length for everyone.
This means that in the back-propagation part, we wont be subtracting the whole output from the corresponding label, rather we only check the index of the correct word in our output against our label(np.argmax() of our label) and see if at that index, our output was correct or not(or simply subtracting that index in our output from 1)

output[0,np.argmax(label)] -= 1


or

output[0,np.argmax(label)] -= label[0,np.argmax(label)]


I also made a mistake in backpropagation section here : instead of using the h_previous, I used dh_previous. so the correct form was :

def rnn_cell_backward(self, xt, h, h_prev, output, true_label, h_gradient_or_dh):
"""
h: is the next state
output: is the output of the current cell!
true_label: is the true label for the current output
h_gradient_or_dh: is the dh which in the beginning is zero
and is updated as we go backward in the backprogagation
remember the backward pass is essentially a loop!
so have that in mind and you will be fine!
"""

e = np.copy(output)
e[0,np.argmax(true_label)] -= true_label[0,np.argmax(true_label)]
per_ts_loss = e[0,np.argmax(true_label)]

# must have shape of W3 which is (vocabsize_or_output_dim_size, hidden_state_size)
dW3 = np.dot(e.T, h)
dh = np.dot(e, self.W3) + h_gradient_or_dh  # from later cell
# dbo +=e.1, we use sum, because we have a batch
#e is a vector, when it is subtracted from label, the result will be added to dbo
# since batchsize is always 1 we dont need to do np.sum(e, axis=0)
dbo = e.reshape(e.shape)

# the input part
dtanh = (1 - h**2) * dh

# compute the gradient of the loss with respect to W1
dxt = np.dot(dtanh, self.W1.T)
# must have the shape of (vocab_size, hidden_state_size)
dW1 = np.dot(xt.T, dtanh)

# compute the gradient with respect to W2
dh_prev = np.dot(dtanh, self.W2)
# shape must be (HiddenSize, HiddenSize)
dW2 = np.dot(h_prev.T, dtanh)

#print('dtanh.shape = ',dtanh.shape)
# dbh += dtanh.1, we use sum, since we have a batch
dbh = np.sum(dtanh, axis=0)

return dW1, dW2, dW3, dbh, dbo, dxt, dh_prev, dh, per_ts_loss


There was also something wrong in the forward pass. Since the sequence length is not the same, we cant have a prespecified H, and O. and thus making them a class field serves nor purpose. Also, we can have 2 options when feeding the h_previous to the calculations.

1. use zero initialized h_previous always
2. use the last hidden state of calculated by previous sample.

Since our examples are not related in anyway, we should use the first option. My first impression was that if we go the second route, we will see the network cant perform decently at all and fails to create meaningful outputs, whereas in the case of the first method, it can quickly learn to produce good outputs. I was wrong!! at least in this example, they create the same exact outputs!
So the rnn_layer_forward should look like this instead :

def rnn_layer_forward(self, Xt, H=None):

batch, input_dim_size, T_x = Xt.shape
# allocate H, and O for the first time
H = np.zeros(shape=(batch, self.hidden_state_size, T_x))
O = np.zeros(shape=(batch, self.outputsize, T_x))

if (self.mode == 1):
h_previous = np.zeros(shape=(batch, self.hidden_state_size))
else:
h_previous = self.H[:, :, -1]

for t in range(T_x):
output, h_t = self.rnn_cell_foward( Xt[:, :, t], h_previous)
self.H[:, :, t] = h_t
self.O[:, :, t] = output

# Our current/new hiddenstate will be the previous hiddenstate for the next round,
h_previous = h_t

return O, H