5
$\begingroup$

It is said that perceptron is linear classifier, but it has a non-linear activation function f = 1 if wx - b >= 0 and f = 0 otherwise

If i will use some non-linear function on linear combination of my data, i think i will get a non-linear classifier. Why it is false?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

It is called a linear classifier because its decision boundary is given by a (linear) hyperplane. Such a hyperplane is given by the set $\{ x | w^tx =b \}$ which thus splits $\mathbf R^n$ into two classes, $\{x | w^tx \leq b\}$ and $\{ x | w^tx > b\}$. You can think of $w$ as the normal vector to this hyperplane and $b$ as an offset by which you shift the hyperplane.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Is a last layer of neural network a linear classifier? $\endgroup$
    – mike
    May 31, 2019 at 8:33
  • $\begingroup$ That depends on your neural network, in general that does not have to be the case. $\endgroup$
    – Stefan
    May 31, 2019 at 8:39
  • $\begingroup$ The hyperplane comment is the correct reason, but the first one isn't sufficient and may not lead to linear classification for certain activation functions $\endgroup$
    – gunes
    May 31, 2019 at 8:46
  • $\begingroup$ @gunes: I think that depends on your definition of "linear classifier". Wikipedia (en.wikipedia.org/wiki/Linear_classifier) defines a linear classifier as a classifier that makes "decision[s] based on the value of a linear combination" of your data. $\endgroup$
    – Stefan
    May 31, 2019 at 8:53
  • 2
    $\begingroup$ @gunes: I see your point, to me the wikipedia article is ambiguous, i.e. it does not rigorously define what a linear classifier is. Having looked up a more reliable source (Wasserman's All of Statistics) I'll agree with you that the correct definition of a linear classifier is "a classifier such that the decision boundary is a hyperplane" and not "a classifier that only depends on a linear combination of the inputs". I'll edit my post accordingly :) $\endgroup$
    – Stefan
    May 31, 2019 at 11:08
0
$\begingroup$

The activation function of a perceptron is linear in it's parameters, i.e. the decision boundary is defined by a linear function of the form $w^tx = b$ = $\sum_{i} w_{i} * x_{i}$.

Similar to linear least squares, the perceptron does not care whether the input $x_{i}$'s are non-linear combinations of your original data. You can indeed model a non-linear decision boundary using a polynomial basis function, e.g. if your data consist of a single independent variable $x$ you could include the square of $x$ as an additional variable.

Again, the perceptron can only model a linear decision boundary, but linear decision boundaries in quadratic space become a non-linear curve when mapped back to the original space.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I dont think this answers this question correctly. While the sumproduct is calculated with linear function, the activation is a nonlinear function (like Sigmoid, ReLU , tan() and others) , so, since sigmoid is the output and it is nonlinear, then the perceptron , even if it is a single one becomes a nonlinear function. If you are saying that perceptron is linear, then show a proof. $\endgroup$
    – Nulik
    Apr 8, 2020 at 8:55
  • $\begingroup$ @Nulik You have a point, but I'm convinced that as long as the activation function is monotonic, choosing a threshold on the activation will lead to the input being separated by a single hyperplane, as there is a bijective mapping between the outputs of the linear model and the non-linear activation. However, if the activation is non-monotonic (I tried this using the activation f(x) = abs(x)), there is a surjective-only mapping of the linear outputs onto the activations, so there are two (or more?) separating hyperplanes in input space, which I'd say would be a non-linear classifier. $\endgroup$
    – Scholar
    Jun 20, 2020 at 13:18
  • $\begingroup$ @Nulik In any case, all separating lines are linear hyperplanes. $\endgroup$
    – Scholar
    Jun 20, 2020 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.