2
$\begingroup$

I'm looking for a general weight update rule for both hidden and output layers, no matter the number of layers, the connections or the transfer function. Does anything like this exist?

I'm quite new to backpropagation and neural networks in general, so if someone has an answer, I would be really grateful if it didn't include too many assumptions about my knowledge in the field.

$\endgroup$
1
$\begingroup$

Here is a nice introduction for you to follow: http://www.mth.kcl.ac.uk/courses-09-10/guide.pdf

$\endgroup$
1
$\begingroup$

None of those factors should change the way you update the weights. All layers are generally updated in the same way.

Here is a basic idea of how to update the weights -

Back propogate the network and find the gradient of each weight with respect to a loss function. This gradient tells you how to change each weight in order to decrease the loss.

Multiply each of these gradients by a small constant (usually called the learning rate), 0.01 for example. This is done because updating each weight by the full gradient is too much and will cause it to explode.

Subtract each new scaled down gradient from each weight.

$\endgroup$
  • 1
    $\begingroup$ Well, that's the basic gradient descent rule for the output layer, but what about the hidden ones? For instance, let's say that in a network with 3 hidden layers I have computed the gradient g of the weight of the connection between the units H3 and O. How can I find the gradient of the connection between H2 and H3, or between H1 and H2, and so on? $\endgroup$ – samuelemarro Feb 3 '16 at 15:46
  • $\begingroup$ You find the gradients for the hidden layers in the same way that you found the output gradients. There is really no difference (just finding the partial derivative of each weight with respect to a loss function). The reason you start with the final layer and work backwards is because when computing the output gradient, you get part of the answer for the final hidden layer's gradients. Then when doing the final layer, you get part of the answer for the layer before it, and so on. $\endgroup$ – Frobot Feb 5 '16 at 4:49
0
$\begingroup$

I copy from here: https://en.wikipedia.org/wiki/Backpropagation but maybe there is some confusion? It says, "backpropagation is a generalization of the (single layer) delta rule, made possible by using the chain rule to iteratively compute gradients for each layer." It is for multilayer(two or more) feed forward networks. If you look into it and try to understand how it works, you can alter it, for example by changing constants in the equation or adding terms that you find sound.

I hope that helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.