-1
$\begingroup$

In order to help myself understand neural networks better, I'm attempting to write the code for a multilayered neural network in Python.

I've written the code for predicting the output, given a set of inputs; however, I'm a bit unsure about how to write the code to do the training.

I know that I need to use the inputs to the matrix in the chain rule to work out the derivative of the weights with respect to the cost of the network.

So at some point, in order to get the value by which to update the weight during training, I will get the connection to which the weight is ascribed, and multiply the input to this connection by an already calculated derivative.

What this means is that for a single set of inputs, I will update the weights using a specific set of values. For another set of inputs, I will need to update the weights with another specific set of values.

Where I am confused is - what do I do when I have multiple sets of inputs?

If I choose to represent the weights of the connections from one layer to another using matrices, does this mean that for each of the multiple inputs, I should do different updates using each of the multiple inputs, meaning that I need to maintain the different sets of weights resulting from each of these different updates? Somehow, this doesn't make sense.

Or do I use all the sets of inputs in my calculations to update the same single set of weights, meaning that I just need to maintain a single set of weights?

$\endgroup$
1
$\begingroup$

Your last sentence is correct, you only maintain a single set of weights.

Let's assume you're using matrices in a two layer network and want to update your weights after every sample. Your inputs will be a vector (a single sample of inputs). Multiply by the first layer's matrix of weights and you have a vector of hidden layer nodes values, to which you apply your activation function. You need to save this until you backpropagate. Then multiply by your second matrix of weights and you have a vector that is your output. When you backpropagate you calculate the derivative of the error and subtract from the second set of weights, then calculate the error on your saved vector of hidden layer node values, take the derivative and subtract from the first set of weights.

Now assume you want to do batch (or mini-batch) training. Your inputs now include many samples, so instead of a vector you have a matrix of input values (one row per sample). Multiply this by the first layer of weights and you have a matrix of node values, which you save until you backpropagate. Multiply by the second layer of weights and you have a matrix of outputs. Calculate the derivative of the error on the entire matrix and divide by the number of rows. In essence, you're updating the weights with the average of all the errors from all the samples in your batch. Do the same thing for the matrix of hidden layer node values to update the first layer of weights with the average of the node value error derivatives.

In either case you always have a single matrix for each layer of weights. What changes is whether you have a vector or a matrix for the inputs, hidden layer node values, and outputs.

$\endgroup$
  • $\begingroup$ Ah. The key in your response is "Calculate the derivative of the error on the entire matrix and divide by the number of rows. In essence, you're updating the weights with the average of all the errors from all the samples in your batch." Thanks! $\endgroup$ – Tola Odejayi Apr 28 '17 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.