# Understanding Weight Updates in a Perceptron

I am learning about perceptrons and how they work. I read that each weight $$w_j$$ is updated based on the equation: $$$$w_j:=w_j+\Delta w_j$$$$

Where:

$$$$\Delta w_j= n \times (y^{(i)}-\hat{y^{(i)}}) \times x_j^{(i)}$$$$

Here, $$n$$ is the learning rate. There are two things that are unclear to me. Namely, why do we take $$x_j^{(i)}$$ (the input) into consideration? Would not this cause problems when different features have different scales? I know standardization is a potential solution but this was not mentioned. Moreover, it would also lead to vastly different weights when a single feature can take a large range of values (example: a pixel with red value 0 and a pixel with red value 255 on the RGB scale). Moreover how are class labels $$\hat{y^{(i)}}$$ represented? Would not choosing numbers 0,1,2... create the same type of problem mentioned above ?

• "Would not this cause problems when different features have different scales?" What problems do you have in mind? Also, why do you think "vastly different weights" would be a problem? Dec 15, 2022 at 3:28

First of all, using the input variable is consistent with the derivative based gradient descent approaches. Other than this, intuitively, perceptron algorithm adds the input variable to the weights such that it tries to make the weights closer/farther to itself. Closer when its label is $$1$$, because the update is $$r(1-\hat y)x$$ and farther when its label is $$0$$, because the update is $$r(0-\hat y)x=-r\hat yx$$. When weight and input vector become closer, their dot product will be larger, and the predicted label will be $$1$$. When they become farther (e.g. in negative directions, $$w=-x\rightarrow$$ $$wx=-||x||^2$$), their dot product will be smaller and the predicted label will be $$0$$. This is why input feature plays an important role on the updates.

This principle still works when input features are of different scales. However, the learning may be slow, or the final result might be different. So, normalization is usually recommended.

In multi-class problems, we have different neurons with binary outputs. The class labels are converted into one-hot vectors and compared with the outputs from these output neurons. The practice of class labels being ordered integers is rare and it also implies an implicit ordering between the classes. Similarly, it can cause problems with the activation function choice.

Your weight update rule is usually called delta rule aka Widrow-Hoff algorithm. Why do we take the input (and some form of output) into consideration to update? This is mainly due to its adaptation based on the famous Hebb's postulate in his The Organization of Behavior:

When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.

Hebb's postulate is simply an unsupervised learning rule for a single neuron and later Widrow et al adapted to your formulated delta rule in order to do supervised learning. So originally there's no concern for data standardization since these inputs and outputs are all neural spikes sharing similar scales.

Finally indeed in NN computation blindly choosing numbers 0,1,2... for class labels could cause issues especially if they're not ordinal but normal categorical values such as gender, where one-hot encoding scheme via dummy variables can be used to alleviate such issue.