I was looking for an intuition for the perceptron algorithm with offset rule, why the update rule is as follows:

cycle through all points until convergence

$\text{if }\, y^{(t)} \neq \theta^{T}x^{(t)} + \theta_0\,\{\\ \quad \theta^{(k+1)} = \theta^{k} + y^{(t)}x^{(t)}\\ \quad\theta^{(k+1)}_0 = \theta^{k}_0 + y^{(t)}\\ \}$

When the offset is zero, I think the update rule is completely intuitive. However, without it, it seems a little odd just adding 1 or -1 to the offset. The only reason I could come up with to explain it was the following but I don't really think its very intuitive explanation and was looking for a different explanation.

My non-intuitive answer:

When the perceptron makes a mistake then:

$y^{(t)}(\theta^{T}x + \theta_0) \leq 0$

But we can re-write the top part as:

$<\theta, \theta_0> \cdot <x^{(t)}, 1> = \theta^{T}x + \theta_0$

and now if we just appeal to the original perceptron rule and change the feature vector to have the one attached at the end and the normal now includes $\theta_0$, now the update would occur as following:

$ \theta'^{(k+1)} = \theta'^{(k)} + y^{(t)}x'^{(t)}$

which is:

$<\theta, \theta_0> + y^{(t)}<x^{(t)}, 1> = <\theta + y^{(t)}x^{(t)}, \theta_0+y^{(t)}>$

I think this might be correct, but even if it is, I didn't really think it was intuitive or "obvious" and was wondering if anyone had a different argument?


PS: Feel free to edit my algorithm to have indentation and spaces, I couldn't make it have indentation without losing the latex :(

  • $\begingroup$ I think your explanation is intuitive enough. Just to rephrase/summarize it, you can think of $\theta_0$ as a parameter associated with an additional coordinate that is set to 1 for all examples. $\endgroup$ – Kristian Georgiev Feb 18 '18 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.