Deriving the line for the decision boundary

Question

Let's say I have the following perceptron activation line: $y = 1$ if $\sum_{i}^{d} w_ix_i \geq \theta $ and $y=0$ else. Now, for d = 4, I have derived the following question. $$ w_4x_4+w_3x_3+w_2x_2+w_1x_1 + w_o$$

I am trying to prove that the distance from the decision boundary to the origin is $ \frac{w^Tx}{||w||}$. I have read the following post. However, I'm not able to understand why we are trying to minimize $x$, which confuses the rest of the reasoning for me. Can someone please elucidate this?

Answer 1

The key observation is that the boundary is exactly

$ w_4x_4+w_3x_3+w_2x_2+w_1x_1 + w_o = 0$,

and that this is exactly the equation of a hyperplane. So the question is how to find the point on this hyperplane that is closest to the origin.

The answer to which you refer solves this particularly elegantly, but, if you prefer, you can bypass it and do it any other number of ways.

For example, the point on this hyperplane with the shortest distance to the origin, is exactly the point on this hyperplane with the shortest square distance to the origin. Using a Lagrange multipliers, therefore, you can optimize

$ \sum_i [x_i^2] $

subject to

$ w_4x_4+w_3x_3+w_2x_2+w_1x_1 + w_0 = 0$,

by solving

$ \sum_i [x_i^2] + \lambda ( w_4x_4+w_3x_3+w_2x_2+w_1x_1 + w_0)$

for the zero derivative.

Differentiating with respect to $x_j$ gives

$2 x_j + \lambda w_j = 0 \Rightarrow x_j = -\frac{\lambda}{2} w_j$ (1).

Differentiating with respect to $\lambda$ gives

$w_4x_4+w_3x_3+w_2x_2+w_1x_1 + w_0 = 0$ (2).

Inserting (1) into (2) gives

$w_0 = \frac{\lambda}{2} \left[w_1^2 + w_2^2 + w_3^2 + w_4^2\right] \Rightarrow \lambda = \frac{2 w_0}{\left[w_1^2 + w_2^2 + w_3^2 + w_4^2\right]}$.

Inserting back into (1) gives

$x_j = \frac{w_j w_0}{\left[w_1^2 + w_2^2 + w_3^2 + w_4^2\right]}$ (3).

The distance to the origin is $\sqrt{x_1^2 + x_2^2 + x_3^2 + x_4^2}$. Inserting (3) into this, gives $\frac{\left| w_0 \right|}{\sqrt{w_1^2 + w_2^2 + w_3^2 + w_4^2}}$