# How are the weights updated in the perceptron learning rule?

I'm considering a perceptron model. I know that when feeding observations from the training dataset to the model, if the model correctly classifies the input, then the weights for this input will not be changed. Conversely, if I feed an observation of the training dataset to the perceptron and the perceptron misclassifies the input, then then the weight will be updated.

The two types of misclassification are when $$\hat y=0$$ and $$y=1$$, and when $$\hat y=1$$ and $$y=0$$, In the first case, is it correct to say that all weights will be increased according to the predetermined learning rate? Similarly, in the second case, all of the weights will be reduced?

• The accepted answer is not really the Perceptrons learning algo. But the weights will not reduce, it depends on the sign of input vector – DuttaA Jun 19 '19 at 19:03

You often define the MSE (the mean squared error) as the loss function of the perceptron. Then you update the weighs using gradient descent and back-propagation (just like any other neural network).

For example, suppose that the perceptron is defined by the weights $$W = (w_1, w_2, w_3)$$, which can initially be zero, and we have the input vector $$X_i = (x_{i1}, x_{i2}, x_{i3})$$, for $$i=1, \dots, n$$, where $$n$$ is the size of the training dataset, which can be defined as $$\mathcal{D} = \{(X_1, t_1), \dots, (X_n, t_n) \}$$. Then, for the $$i$$th pair of the training dataset $$\mathcal{D}$$, the output of the perceptron is defined as a linear combination

$$y_i = W \cdot X_i$$

where $$y_i$$ is the output of the perceptron for the training example $$X_i$$. The loss for the training example $$(X_i, t_i)$$ can be defined as follows

\begin{align} \operatorname{MSE}(X_i, t_i) &= (y_i - t_i)^2\\ &= (W \cdot X_i - t_i)^2\\ &= ((w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) - t_i)^2\\ \end{align}

To update the weights $$W$$, we need to find the partial derivative of $$\operatorname{MSE}(X_i, t_i)$$ with respect to each weight, $$w_1$$, $$w_2$$ and $$w_3$$. The gradient of $$\operatorname{MSE}(X_i, t_i)$$, denoted by $$\nabla_W \operatorname{MSE}(X_i, t_i)$$, that is, the vector containing the partial derivatives of $$\operatorname{MSE}(X_i, t_i)$$ with respect to each element of $$W$$, $$w_1$$, $$w_2$$ and $$w_3$$, can be calculated as follows

\begin{align} \nabla_W \operatorname{MSE}(X_i, t_i) &= \begin{bmatrix} \frac{\partial \operatorname{MSE}(X_i, t_i) }{ w_1}\\ \frac{\partial \operatorname{MSE}(X_i, t_i) }{ w_2}\\ \frac{\partial \operatorname{MSE}(X_i, t_i) }{ w_3} \end{bmatrix}\\ &= \begin{bmatrix} 2((w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) - t_i) \frac{\partial}{w_1} (w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3})\\ 2((w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) - t_i) \frac{\partial}{w_2} (w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3})\\ 2((w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) - t_i) \frac{\partial}{w_3} (w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) \end{bmatrix}\\ &= \begin{bmatrix} 2((w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) - t_i) x_{i1}\\ 2((w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) - t_i) x_{i2}\\ 2((w_1 x_{i1} + w_2 \cdot x_{i2} + w_3 \cdot x_{i3}) - t_i) x_{i3} \end{bmatrix}\\ &= \begin{bmatrix} 2(y_i - t_i) x_{i1}\\ 2(y_i - t_i) x_{i2}\\ 2(y_i - t_i) x_{i3} \end{bmatrix} \end{align}

You can then update the weights of the perceptron using a step of gradient descent

$$W \leftarrow W - \gamma \nabla_W \operatorname{MSE}(X_i, t_i)$$

where $$\gamma$$ is the learning rate. Note that $$W$$ and $$\nabla_W \operatorname{MSE}(X_i, t_i)$$ have the same dimensions.

This shows the application of stochastic gradient descent, that is, you update the weights for every training example $$(X_i, t_i)$$.

You can also perform (batch) gradient descent. You would just define the loss function differently, that is, you would define it as

\begin{align} \operatorname{MSE}(X_i, t_i) &= \frac{1}{n} \sum_{i=1}^n (y_i - t_i)^2\\ \end{align}

Then the derivatives are similarly calculated. To get rid of the $$2$$ in front of the partial derivatives, you can define the loss functions with a $$\frac{1}{2}$$ in front.

The way you change the weights also depends on the outputs. If they are binary (using a threshold-like method to obtain them), which is usually the case, then you can work out what's going to happen when you perform one step of the gradient descent. Just look at the equations above. But, for example, if $$y_i = t_i = 1$$, then $$y_i - t_i = 0$$, then indeed the corresponding weight will not be updated. If $$y_i = 0$$ and $$t_i = 1$$, then $$y_i - t_i = -1$$, the weight will likely be updated (unless the input is zero). So, the updates of the weights also depend on the values of the outputs and targets, that is, you can define the two classes to be $$0$$ and $$1$$ or $$-1$$ and $$1$$ (or something else), and this affects the updates.

• But referring specifically to the perceptron model, isn't the updating of weights much simpler? I know that we could use gradient descent for a multi-layered perceptron model. – Dion Jun 2 '19 at 13:44
• @David AFAIK, this is the usual way of updating the weights (if you use MSE). Have a look at the Wikipedia article (the learning algorithm section): it is analogous to my explanations. This is quite simple anyway, because the back-propagation part is quite simple, compared to a general NN. I showed all steps only for completeness, which makes it look more complicated than it actually is (once you grasp it). – nbro Jun 2 '19 at 13:48
• Ok understood. Could you specifically answer my question though? You may not have seen it because it was part of my edit. – Dion Jun 2 '19 at 19:33
• @David I actually also edited my answer (the last paragraph) to somewho answer your question. If you look at the gradient, its $j$th element is of the form $(y_i - t_i) x_{ij}$. In your case, you are using $\hat{y}$ to denote the output of the perceptron, while I am using $y_i$. Similarly, you're using $y$ to denote the target while I am using $t_i$. Suppose that $y_i = 0$ and $t_i = 1$, then $-1x_{ij}$. Now, you don't know the sign of $x_{ij}$ a priori, so $(y_i - t_i) x_{ij}$ could either be negative or positive, so you could either decrease or increase the corresponding weight. – nbro Jun 2 '19 at 22:21
• So, as I state in the last paragraph of my answer, whether you increase or decrease the weights depends on the inputs, the loss function and how you define the classes. – nbro Jun 2 '19 at 22:22

No it is not necessary for weights to decrease in Perceptron Learning Algorithm. It depends solely on the input vector whether weights will decrease or increase. Since, the Perceptron Learning Algorithm employs the signum function at the output, defining a MSE loss might be an indicator of the loss, but useless for any other purpose nonetheless, accuracy will be equal to MSE loss as $$(y-t_i)^2$$ will be $$+1$$ for mis-calssification, so total mis-classifications are the MSE loss.

As far as your question goes, weights can be multi dimensional, say the classes you want to separate is 2D input vector and the classes lie in 2nd and 4th quadrant respectively. The classifier weights or weight vector $$W$$ will be something like $$[w_1,-w_2]$$ or $$[-w_1,w_2]$$ where $$w_1,w_2>0$$ clearly you can see that if you multiply any point in 2nd quadrant, $$[-x_1^i,x_2^i]$$ where $$i$$ denotes training example (since it is in 2nd quadrant the negative sign), $$x_1>0$$ (x_axis) and $$x_2>0$$ (y_axis), multiply it by the weights, for the first case $$-w_1x_1^i - w_1x_1^i$$ will always be negaitive, and in the second case $$w_1x_1^i + w_1x_1^i$$ always positive. Similarly, for 4th quadrant co-ordinate $$[w_1,-w_2]$$, in the first case $$w_1x_1^i + w_1x_1^i$$ always positive result, and in second case always negative.

So, why are there 2 solutions for weights? Well, clearly the second solution is just $$180^o$$ rotated version of the first weights. Now, you can exceed this to multidimensional cases. Thus, unless all the input values of an input class i.e. all the $$x_i$$ of an input $$[x_1, x_2....x_n]$$ are of the same sign for each of the classes then only all the weight values $$w_i$$ of $$[w_1,...w_n]$$ will move in a certain direction, otherwise you cannot define weight increase or decrease.