Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Consider the perceptron as illustrated in the figure above.
I know:
If the activation function is linear, i.e. the first three cases, then the perceptron is equivalent to a linear classifier.
However, I just wonder:
Is the perceptron equivalent to a non-linear classifier if its activation function is sigmoid?
In other words:
Can the perceptron represent any complex function, if using sigmoid as its activation function?
The network in the diagram has an input layer and an output layer, but no hidden layers. This type of network can't perform nonlinear classification or implement arbitrary nonlinear functions, regardless of the choice of activation function.
The input is projected onto the weight vector and scaled/shifted along this direction. This is a linear operation that reduces the input to a single value, which is then passed through the (possibly nonlinear) activation function. This linear reduction to a single value is the reason the network can't implement arbitrary functions. Consider a hyperplane in input space that's orthogonal to the weight vector. All inputs falling within this hyperplane are mapped to the same output value (the decision boundary plotted below is an example of such a hyperplane).
Here's an example function corresponding to a network with two inputs and a logistic sigmoid output:
The function is a surface bent into a sigmoidal shape along the direction of the weight vector. Changing the network parameters can rotate the direction of the sigmoidal surface, and stretch or shift it. But, the fundamental sigmoidal shape will always remain.
If the network is used to implement a classifier, the decision boundary will always be linear. For example, say we take the output of the example network to represent the probability that the class is '1', given the input (i.e. the network implements a logistic regression model). We impose a threshold such that inputs are classified as '1' if they produce above-threshold output, otherwise '0'. Here's top view of the same function, where color represents the output:
The plotted decision boundary corresponds to a threshold of 0.5 (i.e. we assign the most likely class). All points to the right of this boundary would be classified as '1', and those to the left as '0'. The decision boundary is orthogonal to the weight vector (plotted in red). Changing the weights could rotate or shift the decision boundary, but never make it nonlinear.
However, things change radically once the network contains at least one hidden layer with sigmoidal (or other nonlinear) activation function. Such a network can indeed perform nonlinear classification and approximate arbitrary functions (but doing so may require adding vastly more units to the network). This is a consequence of the universal approximation theorem, as @broncoAbierto mentioned.
I agree with the answer given by @user20160.
However, you may be interested in what happens if the activation function is not monotonic.
For example: bell-shaped gaussian function $g(z) = e^{-v^2/2}$.
In this case, two hyperplanes are generated, which can be used to solve the XOR problem.
To find out more about classification in general, take a look at https://en.wikipedia.org/wiki/Perceptron and https://en.wikipedia.org/wiki/Linear_classifier, for example.
Answering the question, firstly, a classifier does not need to be able to represent any complex function to be considered non-linear. An example is a quadratic classifier (wiki/Quadratic_classifier), which is a polynomial function of degree 2. We can have infinitely many further examples as polynomial classifiers of degree n, n goes from 3 to infinity.
Secondly, no, changing activation function to sigmoid doesn't help if we consider a classical setup. In fact, sigmoid activation function wouldn't even make a sensible classifier. In classical setup the output of perceptron is either -1 or +1, +1 representing Class 1, and -1 representing Class 2. If you changed activation function to sigmoid, you would no longer have an interpretable output. (Now, of course, you can apply a step function after sigmoid, but if you think about it, it is the same as using only the step function)
Clarifying the connection to the broncoAbierto answer, a composition of arbitrarily many perceptrons with sigmoid activation (i.e., a neural network) indeed is a non-linear classifier. Moreover, it can approximate any complex function. A single perceptron, however, doesn't have any of these properties.
You might have heard of the universal approximation theorem for neural networks. Networks with sigmoid activation are a particular case of the networks for which this holds, so yes, they can perform non-linear classification, and they can approximate any function arbitrarily well provided that the number of units is sufficiently large.
