I'm working through AI: A Modern Approach by Russel and Norvig. At the section on back-propagation, they have this to say:

The idea is that hidden node j is responsible for some fraction of the error $\Delta_k$ in each of the output nodes to which it connects, Therefore the $\Delta_k$ values are divided according to the strength of the connection between the hidden node and output node.

and they give this equation: $$ \Delta_j = g'(in_j)\sum_kw_{j,k}\Delta_k $$

That seems wrong to me. Shouldn't the percentage of error $j$ is responsible for be the weight of its connection to the next node $k$ divided by the sum of the weights coming into $k$. making it something like:

$$ \Delta_j = g'(in_j)\sum_k\frac{w_{j,k}}{\sum_iw_{J_i,k}}\Delta_k $$

  • $\begingroup$ @hxd1011, it is best not to add the self study tag for the OP, but to ask them to add it themselves (there is example text here). Then there is a greater chance they will be familiar with our policies. If they don't add the tag, we can close the thread. $\endgroup$ – gung - Reinstate Monica Jul 3 '16 at 23:50
  • $\begingroup$ Please add the [self-study] tag & read its wiki. $\endgroup$ – gung - Reinstate Monica Jul 3 '16 at 23:50
  • $\begingroup$ I feel like it's easier to think about back propagation as just performing gradient descent. The forward pass computes the activations and loss function. The backward pass computes the gradient of the loss function w.r.t. to the parameters, which is just the chain rule. The parameters are updated by stepping in the direction opposite the gradient, scaled by the learning rate. The proper equations just fall out of this picture. $\endgroup$ – user20160 Jul 4 '16 at 9:03

$$ \Delta_j = g'(in_j)\sum_kw_{j,k}\Delta_k $$

is the correct formula. By "divided" they meant "allocated" or "distributed".

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.