Given features x_1...x_n, weights w_1...w_n, calculated output y = Z dot X, and actual output y', the perceptron learning algorithm changes the weights after each iteration as follows:

deltaZ_i = learningRate * (y' - y) * x_i

I understand why we multiply by (y' - y): we want to change the weight to push the calculated output closer to the actual output. Similarly, the sign of x_i has to be taken into account: if x_i is negative, the weight should be shifted in the opposite direction than if it is positive.

What I don't understand is why multiply by the value x_i, rather than just its sign. We multiply by the value of (y' - y) to produce a larger change when the deviation is large, but why produce a larger change when x_i is big in magnitude? We are already multiplying z_i and x_i, so x_i being large is taken into account. It seems that when multiplying deltaZ_i by x_i will cause x_i to be squared in the calculation of y, causing features with large magnitudes to be overrepresented.


migrated from stackoverflow.com Mar 10 '16 at 7:23

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ please use math markup $\endgroup$ – Aksakal Jan 29 '18 at 15:30
  • $\begingroup$ you are not looking for sign and magnitude of the error, but for a slope of the error (derivation) with respect to activation of the neuron. Large relative inputs are a problem, that's why it is a good idea to standardize your inputs before training. $\endgroup$ – rep_ho Jan 29 '18 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.