3
$\begingroup$

Given features $x_1...x_n$, weights $w_1...w_n$, calculated output $y = W^T \cdot X$, and actual output $\hat{y}$, the perceptron learning algorithm changes the weights after each iteration as follows:

$$\Delta z_i = \text{learningRate} \cdot (\hat{y} - y) \cdot x_i$$

I understand why we multiply by $(\hat{y} - y)$: we want to change the weight to push the calculated output closer to the actual output. Similarly, the sign of $x_i$ has to be taken into account: if $x_i$ is negative, the weight should be shifted in the opposite direction than if it is positive.

Question: What I don't understand is why multiply by the value $x_i$, rather than just its sign? We multiply by the value of $(\hat{y} - y)$ to produce a larger change when the deviation is large, but why produce a larger change when $x_i$ is big in magnitude?

We are already multiplying $z_i$ and $x_i$, so $x_i$ being large is taken into account. It seems that when multiplying $\Delta z_i$ by $x_i$ will cause $x_i$ to be squared in the calculation of $\hat{y}$, causing features with large magnitudes to be overrepresented.

$\endgroup$
  • 1
    $\begingroup$ Welcome to our site! This kind of machine learning question is right on-topic here so I've removed your query from the question body; you can look at our help center to see the scope of the site. $\endgroup$ – Silverfish Mar 7 '16 at 23:03
1
$\begingroup$

I'd like to explain a little bit the perceptron.

Let's say the calculated output $h(x)$ is $sign(W^T \cdot X)$, where
enter image description here

Pay attention that the calculated output is the sign(it's 1 if $W^TX \gt 0$ and -1 otherwise) rather than the product value.

We consider only when the classifier makes mistakes, which means that $h(x) - y$(y is the label in observations) is not 0 and $h(x) * y$ is $-1$(the label is 1 but the calculated output is -1, orthe label is -1 but the calculated output is 1).

  1. If $y$ is $-1$ but the prediction is $1$.

enter image description here

The red vector is the incorrect old weight vector and the dotted vector is $yx$. $y$ is $-1$ so it has an opposite direction as $x$, and the prediction, the sign(positive) of the product of $w$ and $x$ with an angle between 0 and 90, is 1.

We update the $w$ by $yx$ and make it the blue vector. After the update the angle of the blue vector(new weights) and the black vector(input) is between 90 and 180, making the sign (which is the same as the label $y$) opposite to that between the red vector $w$ and the black vector $x$

  1. If $y$ is $1$ but the prediction is $-1$.

enter image description here
In this case we need to correct the weight vector to gain an angle with the input vector between 0 and 90 to obtain an inner product that has the same sign of the label(1).
We can apply the same to the old weight vector by adding the negative $yx$.

To sum up, the update always drags the weight vector a little bit to make the sign of inner product has the same sign of the label by add a $yx$(the dotted vector in each case).

Hope I have cleared up the issue a bit and may this explanation help.

EDIT

As described above, the perceptron is the normal perceptron. And an other perceptron which is often called soft perceptron is the same as logistic regression(binomial).

Let me reduce $$\Delta z_i = \text{learningRate} \cdot (\hat{y} - y) \cdot x_i$$
for you from the sigmoid function.

$$h(W^TX) = \frac{1}{1+e^{-W^TX}}$$

\begin{cases} P(y=1; X, W) = h(W^TX) \\ P(y=0; X, W) = 1-h(W^TX) & \end{cases}

We can combine these two into one, $$P(y;X,W)=h(W^TX)^y(1-h(W^TX))^{1-y}$$ And for all samples, $$L(W)=\prod(W^Tx^{(i)})^{y^{(i)}}(1-h(W^Tx^{(i)}))^{(1-y^{(i)})}$$ And we take log of both sides and get, $$l(W) = logL(W)=\sum_{i=1}^my^{(i)}logh(W^Tx^{(i)})+(1-y^{(i)})log(1-h(W^Tx^{(i)}))$$ To get the gradient(or update value) we take the partial derivative with respect to $W$ and get, $$\frac{\partial l}{\partial W}=\sum_{i=1}^m[y^{(i)}-h(W^Tx^{(i)})]x^{(i)}$$ Which is the same as your function.

For the $xy$ case, the activation function is sgn, meaning above a threshhold we take 1 and below it we take -1, but for the latter one, the soft perceptron, the activation is the sigmoid function, thus the gradient is softer. And for the latter one($(\hat y- y)*x$) we are actually evaluating the maximum likelihood.

$\endgroup$
  • $\begingroup$ Thank you for the nice graphics and clear explanation. The question asked why can't we just take the sign of $x$. So can you draw the same figures but where $w$ is updated as $w+y\text{ sign}(x)$ rather than $w+yx$? And then see why it may or may not work? $\endgroup$ – highBandWidth Mar 21 '17 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.