1
$\begingroup$

I've been following an algorithm described on a book called Knowledge Discovery with Support Vector Machines by Lutz H. Hamel. In the book, there is this learning algorithm for a single perceptron below.

$$ \begin{align*} &\textbf{let} \quad D = \{(\bar{x_1}, y_1), (\bar{x_2}, y_2), ...,(\bar{x_l}, y_l)\} \subset \mathbb{R^n} x \{+1, -1\} \\ &\textbf{let} \quad 0 < \eta < 1 \\ &\bar{w} \leftarrow \bar{0}\\ &b \leftarrow 0 \\ &r \leftarrow max\{|\bar{x}|\, |\, (\bar{x}, y) \in D\ \}\\ &\textbf{repeat}\\ &\quad\textbf{for} \,i = 1\, \textbf{to}\, l\,\\ &\quad\quad \textbf{if}\, sgn(\bar{w}\cdot\bar{x_i} -b) \neq y_i\, \textbf{then}\\ &\quad\quad\quad\bar{w}\leftarrow\bar{w} +\eta y_i \bar{x_i}\\ &\quad\quad\quad b \leftarrow b - \eta y_i r^2\\ &\quad\quad\textbf{endif}\\ &\quad\textbf{endfor}\\ &\textbf{until}\, sgn(\bar{w}\cdot\bar{x_j}-b) = y_j \, with j=1, 2, ...,l\\ &\textbf{return}\, (\bar{w}, b) \end{align*} $$

So our decision boundary will be the line $\bar{w}\cdot\bar{x}=b$

Which can be written as,

$$ w_1 x_1 + w_2 x_2 - b = 0\quad\quad (1)\\ x_2 = -\frac{w_1}{w_2} x_1 + \frac{b}{w_2}\quad\quad(2) $$


Everything is fine until here and I want to animate the training of the perceptron using manim. Which will look something like this:

manim-gif

What I don't understand is, let's say I want to start my training with

$$ w = \begin{bmatrix} 0\\ 0 \end{bmatrix}\\ b = 0 $$

in this case, how can I draw the decision boundary? Because if we just substitute the values in equation (2) we get:

$$ x_2 = -\frac{0}{0}x_1 + \frac{b}{0} $$

  • What's my decision boundary in this case?
  • I've got another question about the GIF above. Should I draw the position vectors of the samples and the weight relative to the decision boundary and not the origin?
$\endgroup$
1
  • 1
    $\begingroup$ Regarding your first question, this is the reason why we usually sample the initial parameter values from a normal or uniform distribution. $\endgroup$ – mhdadk Jan 4 at 16:23
4
$\begingroup$

TL;DR: Your decision boundary is the whole $(x_1, x_2)$ plane.

In detail: The function

$$ z = w_1 x_1 + w_2 x_2 - b $$

is a plane in the 3D space, spanned by the axes $(x_1, x_2, z)$. Where $z = 0$, the plane intersects the horizontal plane, spanned by $(x_1, x_2)$. The intersection of these two planes in most cases happens to be a straight line, which we, in this context, call the 'class boundary'.

But, in general, the class boundary is the set of points where $z = 0$. In your special case, with $w_1 = w_2 = b = 0$, this is everywhere, regardless of the values $x_1, x_2$. So, the class boundary is the whole $(x_1, x_2)$ plane.

P.S. I'm impressed by your graphics.

$\endgroup$
1
  • $\begingroup$ Now I get it. Maybe I could then color the whole plane to show that when the weights are 0, the decision boundary is the plane itself. Thank you for the explanation and for the kind words, I've just used the library I've linked to, to animate. :) $\endgroup$ – André Yuhai Jan 4 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.