I'm currently implementing a small neural network library to learn the concepts coming with them better. As probably most people do, I'm struggling a bit with the backpropagation algorithm, especially for multiple hidden layers.
My understanding of the algorithm: We can calculate how much each weight of each layer adds to the error by applying the chain rule backwards through the net.
We can calculate the derivatives at each layer from the output of the layer, the delta of the next layer, and the weights of the next layer.
I've found a formula for this online, the code looks like that: For the output layer:
def backward(self, output, y): error = y.T - output delta = error * self.activation_function(output, derivative = True)
And for the layer before the output:
def backward(self, output, gradients_next_layer, weights_next_layer): error = self.gradients_next_layer.dot(weights_next_layer.T) delta = error*self.activation_function(output, derivative =True) return error, delta
True flag in the
activation_function makes it compute the derivative.
This code is conceptually taken from here.
Now my main question would be, if the formula would be exactly the same for a second hidden layer? If my
backward function for the hidden layer is correct, can I apply it like that to a second hidden layer?
For example, here would be my
def backprop(self,X,y,output): self.output_error, self.output_delta = self.output_layer.backward(output, y) self.z2_error, self.z2_delta = self.hidden_layer_2.backward(self.z2, self.output_delta, self.output_layer.weights) self.z_error, self.z_delta = self.hidden_layer_1.backward(self.z, self.z2_delta, self.hidden_layer_2.weights) self.hidden_layer_1.weights += self.lr * X.T.dot(self.z_delta) self.hidden_layer_1.bias += self.lr * np.sum(self.z_delta, axis=0,keepdims=True) self.hidden_layer_2.weights += self.lr * self.z.T.dot(self.z2_delta) self.hidden_layer_2.bias += self.lr * np.sum(self.z2_delta, axis=0,keepdims=True) self.output_layer.weights += self.lr * self.z.T.dot(self.output_delta) self.output_layer.bias += self.lr * np.sum(self.output_delta, axis=0,keepdims=True)
X is the training batch and
y are the gold truth labels for the batch.
I know of the vanishing gradients problem when multiple hidden layers are chained. I mainly care about if my understanding of the procedure is correct at the moment.
Appreciate any hints.