It is said that perceptron is linear classifier, but it has a non-linear activation function f = 1 if wx - b >= 0 and f = 0 otherwise

If i will use some non-linear function on linear combination of my data, i think i will get a non-linear classifier. Why it is false?


It is called a linear classifier because its decision boundary is given by a (linear) hyperplane. Such a hyperplane is given by the set $\{ x | w^tx =b \}$ which thus splits $\mathbf R^n$ into two classes, $\{x | w^tx \leq b\}$ and $\{ x | w^tx > b\}$. You can think of $w$ as the normal vector to this hyperplane and $b$ as an offset by which you shift the hyperplane.

  • $\begingroup$ Is a last layer of neural network a linear classifier? $\endgroup$ – mike May 31 '19 at 8:33
  • $\begingroup$ That depends on your neural network, in general that does not have to be the case. $\endgroup$ – Stefan May 31 '19 at 8:39
  • $\begingroup$ The hyperplane comment is the correct reason, but the first one isn't sufficient and may not lead to linear classification for certain activation functions $\endgroup$ – gunes May 31 '19 at 8:46
  • $\begingroup$ @gunes: I think that depends on your definition of "linear classifier". Wikipedia (en.wikipedia.org/wiki/Linear_classifier) defines a linear classifier as a classifier that makes "decision[s] based on the value of a linear combination" of your data. $\endgroup$ – Stefan May 31 '19 at 8:53
  • 1
    $\begingroup$ @gunes: I see your point, to me the wikipedia article is ambiguous, i.e. it does not rigorously define what a linear classifier is. Having looked up a more reliable source (Wasserman's All of Statistics) I'll agree with you that the correct definition of a linear classifier is "a classifier such that the decision boundary is a hyperplane" and not "a classifier that only depends on a linear combination of the inputs". I'll edit my post accordingly :) $\endgroup$ – Stefan May 31 '19 at 11:08

The activation function of a perceptron is linear in it's parameters, i.e. the decision boundary is defined by a linear function of the form $w^tx = b$ = $\sum_{i} w_{i} * x_{i}$.

Similar to linear least squares, the perceptron does not care whether the input $x_{i}$'s are non-linear combinations of your original data. You can indeed model a non-linear decision boundary using a polynomial basis function, e.g. if your data consist of a single independent variable $x$ you could include the square of $x$ as an additional variable.

Again, the perceptron can only model a linear decision boundary, but linear decision boundaries in quadratic space become a non-linear curve when mapped back to the original space.

  • $\begingroup$ I dont think this answers this question correctly. While the sumproduct is calculated with linear function, the activation is a nonlinear function (like Sigmoid, ReLU , tan() and others) , so, since sigmoid is the output and it is nonlinear, then the perceptron , even if it is a single one becomes a nonlinear function. If you are saying that perceptron is linear, then show a proof. $\endgroup$ – Nulik Apr 8 '20 at 8:55
  • $\begingroup$ @Nulik You have a point, but I'm convinced that as long as the activation function is monotonic, choosing a threshold on the activation will lead to the input being separated by a single hyperplane, as there is a bijective mapping between the outputs of the linear model and the non-linear activation. However, if the activation is non-monotonic (I tried this using the activation f(x) = abs(x)), there is a surjective-only mapping of the linear outputs onto the activations, so there are two (or more?) separating hyperplanes in input space, which I'd say would be a non-linear classifier. $\endgroup$ – Scholar Jun 20 '20 at 13:18
  • $\begingroup$ @Nulik In any case, all separating lines are linear hyperplanes. $\endgroup$ – Scholar Jun 20 '20 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.