I am learning about artificial neural networks. I understand how the weights determine the slope of the (orthogonal) decision boundary and how the bias shifts that decision boundary, much like a line. What I do not understamd is why the equation is always set to zero?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
That's linear algebra, not specific to perceptrons or machine learning. Let's start with a simple, two-dimensional case, with axes $x_1$ and $x_2$. Your class boundary can be defined by a straight line:
$$x_2 = a x_1 + b$$
This is a so-called explicit line equation. It has one problem: What if the boundary is a vertical line? In that case $a$ would need to be infinte. You can circumvent this problem by swapping the coordinates and writing:
$$x_1 = c x_2 + d$$
A vertical line is then described by $x_1 = d$, i.e. taking $c=0$. However, here you have the same problem with the horizontal line, because $c$ would have to be infinite. In the first equation, the horizontal line would be written down as $x_2 = b$, i.e. setting $a$ to zero.
The implicit equation, which is easy to derive from the above two, unifies the notation into a general form and is not susceptible to that problem:
$$w_1 x_1 + w_2 x_2 + b = 0$$
You can get the first explicit equation from it by setting $w_1 = a$ and $w_2 = -1$, and the second equation by setting $w_1 = -1$, $w_2 = c$, and $b=d$. A horizontal line can be written down by setting $w_1 = 0$, and a vertical one by setting $w_2 = 0$.
Using vector notation, with $w = [w_1, w_2]$ and $x = [x_1, x_2]$, the implicit equation can be written more compactly as:
$$w \cdot x + b = 0$$
This can be generalized to any number of dimensions.
-
$\begingroup$ The simplified case helps me understand a bit more, but can you clarify how the implicit equation is not susceptible to the infinite a problem? Is it not possible to have a verticle line as a decision boundary? or is it becuase the if the wieght is infinite then the decision boundary is horizontal? $\endgroup$ Commented Jan 29, 2020 at 18:46
-
$\begingroup$ I edited the answer and hope it to be clearer now. $\endgroup$– Igor F.Commented Jan 30, 2020 at 9:28
-
$\begingroup$ Could you please elaborate more on the explicit line equations? I couldn't get why a would need to be infinite. Is it because a would need to compensate for all the different values of $x_2$ regardless of $b$ being any arbitrary value? $\endgroup$ Commented Dec 21, 2020 at 10:20
-
1$\begingroup$ @AndréYuhai $a$ is the slope of the line. It gives you the change in $x_2$ for a unit change in $x_1$. For $a=1$, if you change $x_1$ by 1, $x_2$ will also change by 1. That's a 45° line. As $a$ grows, the line becomes steeper. For $a=1000$, increasing $x_1$ by 1 causes an increase of $x_2$ by 1000. That's already much steeper, but still not exactly vertical. Only when $a = \infty$, the line becomes vertical. It's easy to understand if you draw it up. $\endgroup$– Igor F.Commented Dec 22, 2020 at 9:54
-
$\begingroup$ Oh, now I got it! Thank you! $\endgroup$ Commented Dec 22, 2020 at 13:08
$\begingroup$
$\endgroup$
The equation is parametrizing a hyperplane.
You could set your decision boundary to be any parametrized by say Wp + b' = 1 if you wanted to. However, the results would be the same. Since this would be equal to saying the decision boundary is Wp + b'-1 = 0, and then we can define b=b'-1 and estimate b instead of b'
