3
$\begingroup$

I was reading a book named Support Vector Machines Succinctly written by Alexandre Kowalczyk.

In first chapter, Hyperplane section:

I understood how did i end up to the hyperplane equation: w.x + b = 0 Now, logically if it is an equation of hyperplane then it is true that all points lies on hyperplane must satisfy above equation.

Now, We can see the relationship between line equation and hyperplane equation. Let vector w = (w0, w1) and x = (x, y) and b, we define a hyperplane having equation:

w.x + b = 0 This is equivalent to: w0x + w1y + b =0 is equal to w1y = - w0x - b

We isolate y to get: y = -(w0 / w1)x - (b / w1)

if we define a and c: a = -(w0 / w1) and c = -(b / w1)

Then we get, y = ax + c

So, here bias c of line equation is equal to bias b of hyperplane equation when w1 = -1.

Immediate above line I wrote exactly from that book. I don't understand why they calling b or c as a bias? They were supposed to be call intercept. I don't know what is bias mean here.

Now, from hyperplane equation they got a function called as hypothesis function: h(xi) = sign(w.xi +b). The job of this function is to predict label of a given data x.

Again, as this hypothesis function use hyperplane equation which produced linear combination of the values, this function is also called linear classifier.

Same equation have three different names: hyperplane, hypothesis function and linear classifier. I don't understand why they are using three different names?

$\endgroup$

1 Answer 1

3
$\begingroup$

Immediate above line I wrote exactly from that book. I don't understand why they calling b or c as a bias? They were supposed to be call intercept. I don't know what is bias mean here.

The word "bias" in this context is the same as the intercept.

Same equation have three different names: hyperplane, hypothesis function and linear classifier. I don't understand why they are using three different names?

The answer to this question is a bit more subtle. Based on your question, I assume you are using an SVM for classification. For simplicity, suppose you have two classes: $c_1$ and $c_2$. Also, suppose that all examples you want to classify are represented as feature vectors, where an individual feature vector is denoted as $\mathbf{x}$. The "best" classifier is the Bayes classifier, which states the following classification rule:

$$ \mathbf{x} \in c_1 \ \ \text{if} \ \ p(c_1|\mathbf{x}) > p(c_2|\mathbf{x}) \\ \mathbf{x} \in c_2 \ \ \text{if} \ \ p(c_1|\mathbf{x}) < p(c_2|\mathbf{x}) $$

In other words, assign $\mathbf{x}$ to class $c_1$ if the probability of $c_1$ given $\mathbf{x}$ is larger than the probability of $c_2$ given $\mathbf{x}$, and assign $\mathbf{x}$ to class $c_2$ otherwise. Since:

$$ \begin{align} p(c_1|\mathbf{x}) + p(c_2|\mathbf{x}) &= 1 \\ p(c_2|\mathbf{x}) &= 1 - p(c_1|\mathbf{x}) \end{align} $$

Then we only need to compute $p(c_1|\mathbf{x})$ to perform Bayes classification. Now notice that:

$$ \begin{align} p(c_1|\mathbf{x}) &= \frac{p(\mathbf{x}|c_1)p(c_1)}{p(\mathbf{x})} \\ &= \frac{p(\mathbf{x}|c_1)p(c_1)}{p(\mathbf{x}|c_1)p(c_1) + p(\mathbf{x}|c_2)p(c_2)} \end{align} $$

Dividing the numerator and denominator by $p(\mathbf{x}|c_1)p(c_1)$ yields:

$$ p(c_1|\mathbf{x}) = \frac{1}{1 + \frac{p(\mathbf{x}|c_2)p(c_2)}{p(\mathbf{x}|c_1)p(c_1)}} $$

Since:

$$ \frac{p(\mathbf{x}|c_2)p(c_2)}{p(\mathbf{x}|c_1)p(c_1)} = \exp\left(\text{ln}\left(\frac{p(\mathbf{x}|c_2)p(c_2)}{p(\mathbf{x}|c_1)p(c_1)}\right)\right) $$

Then:

$$ \begin{align} p(c_1|\mathbf{x}) &= \frac{1}{1 + \exp\left(\text{ln}\left(\frac{p(\mathbf{x}|c_2)p(c_2)}{p(\mathbf{x}|c_1)p(c_1)}\right)\right)} \\ &= \sigma\left(\text{ln}\left(\frac{p(\mathbf{x}|c_1)p(c_1)}{p(\mathbf{x}|c_2)p(c_2)}\right)\right) \end{align} $$

Where $\sigma(\cdot)$ is the logistic function. Therefore, to compute $p(c_1|\mathbf{x})$, we need to compute $p(\mathbf{x}|c_1),p(c_1),p(\mathbf{x}|c_2),$ and $p(c_2)$. However, in practice, it is usually difficult to estimate $p(\mathbf{x}|c_1)$ and $p(\mathbf{x}|c_2)$. We could instead try to estimate $p(c_1|\mathbf{x})$ directly.

Notice that the expression:

$$ \text{ln}\left(\frac{p(\mathbf{x}|c_1)p(c_1)}{p(\mathbf{x}|c_2)p(c_2)}\right) $$

Is only a function of $\mathbf{x}$. Therefore, let:

$$ \text{ln}\left(\frac{p(\mathbf{x}|c_1)p(c_1)}{p(\mathbf{x}|c_2)p(c_2)}\right) = f(\mathbf{x};\theta) $$

Where the function $f$ is parameterized by $\theta$ such that:

$$ p(c_1|\mathbf{x}) = \sigma(f(\mathbf{x};\theta)) $$

Finally, notice that the logistic function looks like this:

enter image description here

In other words, for positive inputs, its output is greater than 0.5, and for negative inputs, its output is less than 0.5. Therefore, we don't actually need to compute:

$$ p(c_1|\mathbf{x}) = \sigma(f(\mathbf{x};\theta)) $$

We can just check the sign of the hypothesis function $f(\mathbf{x};\theta)$, since if it's positive, then we know that $p(c_1|\mathbf{x})$ is greater than 0.5, and since:

$$ p(c_1|\mathbf{x}) + p(c_2|\mathbf{x}) = 1 $$

Then we know that $p(c_2|\mathbf{x})$ is less than 0.5, and so we assign $\mathbf{x}$ to class $c_1$.

One way to choose the hypothesis function $f(\mathbf{x};\theta)$ is a linear function of $\mathbf{x}$:

$$ f(\mathbf{x};\mathbf{w},b) = \mathbf{w}^T\mathbf{x} + b $$

Where $\theta = (\mathbf{w},b)$. Now, the Bayes classification rule becomes:

$$ \mathbf{x} \in c_1 \ \ \text{if} \ \ \text{sign}(\mathbf{w}^T\mathbf{x} + b) = +1 \\ \mathbf{x} \in c_2 \ \ \text{if} \ \ \text{sign}(\mathbf{w}^T\mathbf{x} + b) = -1 $$

This is the definition of a linear classifier. You get different classifiers based on your choice of the hypothesis function $f(\mathbf{x};\theta)$.

The term hyperplane in this context is just the multi-dimensional plane that separates the two classes of feature vectors in the space of all possible feature vectors, also known as feature space. Since the $\text{sign}(\cdot)$ function switches classes when its input is 0, then the hyperplane equation is:

$$ \mathbf{w}^T\mathbf{x} + b = 0 $$


There is a whole field that studies the choice of the hypothesis function $f(\mathbf{x};\theta)$ called statistical learning theory.

$\endgroup$
2
  • $\begingroup$ Since the sign(⋅) function switches classes when its input is 0, then the hyperplane equation is: 𝐰𝑇𝐱+𝑏=0 I don't understand this part $\endgroup$
    – F.C. Akhi
    Feb 21, 2021 at 16:03
  • $\begingroup$ When the result of $\mathbf{w}^T\mathbf{x}+b$ is positive for some $\mathbf{x}$, then $\mathbf{x}$ is assigned to class $c_1$, and when its result is negative for another $\mathbf{x}$, then $\mathbf{x}$ is assigned to class $c_2$. Therefore, as you go from positive to negative, the resulting class goes from $c_1$ to $c_2$. What is the point at which you change from positive to negative? It's when $\mathbf{w}^T\mathbf{x}+b=0$. $\endgroup$
    – mhdadk
    Feb 21, 2021 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.