Do neural networks create a separating hyperplane?

Question

https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ mentions that neural networks learn a representation of the data so as to make the classes linearly separable. What I fail to see is how does a neural network create a separating hyperplane that separates the classes. How do we know that it creates a hyperplane? What's the math behind it?

Answer 1

Disclaimer: I assume your question refers to multi-layer perceptrons, used as classifiers. But, if you want to engage in hair splitting, there are other neural network architectures and applications which do not create a separating hyperplane.

In a multi-layer perceptron, each neuron first computes a linear function on its inputs and then passes it through an activation function, which is almost always non-linear. All layers except the last are used to perform some kind of non-linear data transformation, with the eventual purpose of making the data linearly separable, or close to linearly separable. The neurons in the final, output layer also compute a linear function of their inputs. Typically, they would also pass it through a non-linear function, like a sigmoid, but this is not necessary for understanding the concept. It suffices to set a threshold on the linear output of the output neuron, and you get a separating hyperplane.

Mathematically: Let us denote by $x_i$ the output of the $i$-th neuron in the last hidden layer, i.e. the last layer before the output layer. Let there be just one output neuron in the output layer and let $w_i$ be the weight connecting the $i$-th neuron to the output neuron. Also, let $w_0$ be the bias of the output neuron. Then, the output neuron computes the linear function:

$$ y = w_0 + \sum_i w_i x_i $$

Let us say, we classify all observations for which $y < 0$ as "class A" and the others as "class B". In other words, the equation

$$ w_0 + \sum_i w_i x_i = 0 $$

describes the class boundary. This is again a linear function; an equation describing a hyperplane in the implicit form.

Example: Let's say you have two neurons in the last hidden layer, with the outputs $x_1$ and $x_2$. Then, the equation

$$ w_0 + w_1 x_1 + w_2 x_2 = 0 $$

describes a one-dimensional class boundary (a straight line) in the two-dimensional space, spanned by $x_1$ and $x_2$. If you want, you can reformat the equation to get a perhaps more common explicit form:

$$ x_2 = -\frac{1}{w_2}(w_0 + w_1 x_1) $$

The same works for any number of dimensions.

---

Update in response to comments:

The above described network, with only one neuron in the output layer, is suitable only for classifying the data into two classes. For more classes, you'll need more output neurons. In practice, people usually use as many output neurons as they have classes. Each output neuron is "responsible" for recognising one class. It should output a high value if it "thinks" that the present observation belongs to "its" class, and a low value otherwise. Again, each output neuron determines a separating hyperplane, with points belonging to "its" class ideally on one, positive, high-valued side of the hyperplane and all other points on the other.

Of course, in such case it is possible for the network to produce ambiguous results, e.g. two or more neurons claiming that a point belongs to their class. Resolving these ambiguities is beyond the scope of this question, but the basic idea is to trust more the neuron with the highest value. People commonly use some kind of normalisation, like softmax, to convert the network output to "class probabilities".

As noted above, it is also common for the output neurons to apply a non-linear transformation to the previously computed linear function. Usual choices are $\tanh$ and logistic function, which are both sigmoid, meaning that they approach horizontal asymptotes as their arguments approach positive or negative infinity. With a suitable encoding of class labels as the values of these asymptotes, the training of the neural network consists of adjusting the weights so that its output, according to some error metric, approaches the true class labels.

Answer 2

The final layer can be seen as a generalized linear model (such a a logistic regression) on extracted features.

Consider a basic network with an input layer, a hidden layer, and an output layer. Draw it out and then cover up the input layer. Now it looks like a linear or generalized linear model (such as logistic regression) on some features.

That’s what’s happening and what the author means: the network uses early layers to figure out the features. Then the network regresses on those features.

Answer 3

As the other answers have stated, all the layers up to the last layer can be viewed as a complicated way to create features (feature map), with the last layer representing a linear model. The traditional neural network construction with sigmoid activation function corresponds nicely with logistic regression.

The power of neural networks comes from the nonlinearity of the activation function, since stacking linear layers is equivalent to just one linear layer. Theoretical results are various universal approximation theorems, the first by Cybenko (1989) stating that arbitrary width neural networks with only one hidden layer can approximate any continuous function arbitrarily well. However this is an existence result and does not indicate how to actually learn these weights.

More recently (2018), neural tangent kernels (NTKs) describe the learning process of gradient descent in NNs when considering the limit of infinite width. However, as a kernel method, in the unrealistic infinite-width regime this simplifies to a linear model and does not match up with empirical performance of over-parameterized neural networks (Arona et al, 2019).

Answer 4

I think I may have answered my question.....

Let $x$ be the last hidden layer before the output layer in the neural network
Say I have $n$ classes
Let the multi-class decision rule be simply the class with the highest output value
Mathematically, we classify $x$ as belonging to a class $i$ if $\forall$$j$:
$f(W_ix+b_i) > f(W_{j\neq i}x+b_j)$ where $f$ is a nonlinear activation.
We now solve for the values of $x$ which satisfy this decision rule (i.e. $\{x|decision(x) = i\}$):
If $f$ is monotonically increasing this implies $f(W_ix+b_i) > f(W_{j\neq i}x +b_j) \iff W_ix + b_i > W_{j\neq i}x+b_j$
bringing everything to one side gives:
$\implies (W_{i1}-W_{j1})x_1+(W_{i2}-W_{j2})x_2+(W_{i3}-W_{j3})x_3 +...+ (b_i -b_j) > 0$
so this decision rule does correspond to a separating hyperplane
Edit: it's a hyperplane for fixed $j$, but the inequality must hold for all $j$, so I guess it must be the region that's in common/intersection of all of the hyperplanes.....?

It would be interesting to obtain an explicit expression and solve for the set $x$ that satisfies a decision rule where instead of $x$ being the layer before the output layer, it instead would be the input to a multi-layer neural network