# Do neural networks create a separating hyperplane?

https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ mentions that neural networks learn a representation of the data so as to make the classes linearly separable. What I fail to see is how does a neural network create a separating hyperplane that separates the classes. How do we know that it creates a hyperplane? What's the math behind it?

Disclaimer: I assume your question refers to multi-layer perceptrons, used as classifiers. But, if you want to engage in hair splitting, there are other neural network architectures and applications which do not create a separating hyperplane.

In a multi-layer perceptron, each neuron first computes a linear function on its inputs and then passes it through an activation function, which is almost always non-linear. All layers except the last are used to perform some kind of non-linear data transformation, with the eventual purpose of making the data linearly separable, or close to linearly separable. The neurons in the final, output layer also compute a linear function of their inputs. Typically, they would also pass it through a non-linear function, like a sigmoid, but this is not necessary for understanding the concept. It suffices to set a threshold on the linear output of the output neuron, and you get a separating hyperplane.

Mathematically: Let us denote by $$x_i$$ the output of the $$i$$-th neuron in the last hidden layer, i.e. the last layer before the output layer. Let there be just one output neuron in the output layer and let $$w_i$$ be the weight connecting the $$i$$-th neuron to the output neuron. Also, let $$w_0$$ be the bias of the output neuron. Then, the output neuron computes the linear function:

$$y = w_0 + \sum_i w_i x_i$$

Let us say, we classify all observations for which $$y < 0$$ as "class A" and the others as "class B". In other words, the equation

$$w_0 + \sum_i w_i x_i = 0$$

describes the class boundary. This is again a linear function; an equation describing a hyperplane in the implicit form.

Example: Let's say you have two neurons in the last hidden layer, with the outputs $$x_1$$ and $$x_2$$. Then, the equation

$$w_0 + w_1 x_1 + w_2 x_2 = 0$$

describes a one-dimensional class boundary (a straight line) in the two-dimensional space, spanned by $$x_1$$ and $$x_2$$. If you want, you can reformat the equation to get a perhaps more common explicit form:

$$x_2 = -\frac{1}{w_2}(w_0 + w_1 x_1)$$

The same works for any number of dimensions.

The above described network, with only one neuron in the output layer, is suitable only for classifying the data into two classes. For more classes, you'll need more output neurons. In practice, people usually use as many output neurons as they have classes. Each output neuron is "responsible" for recognising one class. It should output a high value if it "thinks" that the present observation belongs to "its" class, and a low value otherwise. Again, each output neuron determines a separating hyperplane, with points belonging to "its" class ideally on one, positive, high-valued side of the hyperplane and all other points on the other.

Of course, in such case it is possible for the network to produce ambiguous results, e.g. two or more neurons claiming that a point belongs to their class. Resolving these ambiguities is beyond the scope of this question, but the basic idea is to trust more the neuron with the highest value. People commonly use some kind of normalisation, like softmax, to convert the network output to "class probabilities".

As noted above, it is also common for the output neurons to apply a non-linear transformation to the previously computed linear function. Usual choices are $$\tanh$$ and logistic function, which are both sigmoid, meaning that they approach horizontal asymptotes as their arguments approach positive or negative infinity. With a suitable encoding of class labels as the values of these asymptotes, the training of the neural network consists of adjusting the weights so that its output, according to some error metric, approaches the true class labels.

• So I guess it's about having a threshold, but don't neural networks classify according to which of the output neurons have the greatest probability? In this case, is it correct to say that the threshold in this case would be 1/n, where n is the number of classes? I'm a bit confused what effect a non-linear activation function has on this, so if you classify based on whether it's greater than a threshold (after applying non-linear activation) should also correspond to a threshold before applying the non-linear activation, the activation must then be monotonically increasing for this to work?
– JJP7
Commented Aug 24, 2022 at 9:27
• So if you have multiple output neurons, then for each output neuron, there corresponds it's own hyperplane that classifies whether it's in this class or not? If you have two output neurons/two classes those two hyperplanes are the same?
– JJP7
Commented Aug 24, 2022 at 9:32
• Re: Threshold: Yes, the activation function must be monotonic. Typically $\tanh$ or logistic, in which case you can simply compare the output to $0$ or $0.5$, respectively. It is, however, unusual to use just a single neuron to classify $n$ classes, as this would imply a certain structure among the classes, which is very unnatural. Instead, you'd typically have $n$ output neurons, one for each class ($n-1$ would technically work, too, but $n$ is simpler to interpret for more than two classes). Commented Aug 24, 2022 at 10:13
• Re: Multiple neurons: Exactly, each output neuron would define its own separating hyperplane, with, in case of a perfect separation, members of one class on one side and members of all other on the other side. For two classes you could use two output neurons, but their output would be redundant, either identical or complementary. For more than two classes, it is common to use as many output neurons as classes. Commented Aug 24, 2022 at 10:16
• say I have 4 classes, and I'm using softmax, I could have output values of [0.45, 0.30, 0.15, 0.10], I thought that having a threshold of 1/4=0.25 would work, but in this case it would classify the point as belonging to two classes. Also as in youtu.be/Ilg3gGewQ5U?t=75 values may not sum up to 1, also in this case, a fixed threshold would not work, as you can have outputs[0.1, 0.01, 0.01, 0.01]aslongasone of them is greater than the others.Classification is not dictated by a threshold value but by how a value compares with other values does this make the hyperplane interpretation fail?
– JJP7
Commented Aug 24, 2022 at 13:14

The final layer can be seen as a generalized linear model (such a a logistic regression) on extracted features.

Consider a basic network with an input layer, a hidden layer, and an output layer. Draw it out and then cover up the input layer. Now it looks like a linear or generalized linear model (such as logistic regression) on some features.

That’s what’s happening and what the author means: the network uses early layers to figure out the features. Then the network regresses on those features.

As the other answers have stated, all the layers up to the last layer can be viewed as a complicated way to create features (feature map), with the last layer representing a linear model. The traditional neural network construction with sigmoid activation function corresponds nicely with logistic regression.

The power of neural networks comes from the nonlinearity of the activation function, since stacking linear layers is equivalent to just one linear layer. Theoretical results are various universal approximation theorems, the first by Cybenko (1989) stating that arbitrary width neural networks with only one hidden layer can approximate any continuous function arbitrarily well. However this is an existence result and does not indicate how to actually learn these weights.

More recently (2018), neural tangent kernels (NTKs) describe the learning process of gradient descent in NNs when considering the limit of infinite width. However, as a kernel method, in the unrealistic infinite-width regime this simplifies to a linear model and does not match up with empirical performance of over-parameterized neural networks (Arona et al, 2019).

I think I may have answered my question.....

Let $$x$$ be the last hidden layer before the output layer in the neural network
Say I have $$n$$ classes
Let the multi-class decision rule be simply the class with the highest output value
Mathematically, we classify $$x$$ as belonging to a class $$i$$ if $$\forallj$$:
$$f(W_ix+b_i) > f(W_{j\neq i}x+b_j)$$ where $$f$$ is a nonlinear activation.
We now solve for the values of $$x$$ which satisfy this decision rule (i.e. $$\{x|decision(x) = i\}$$):
If $$f$$ is monotonically increasing this implies $$f(W_ix+b_i) > f(W_{j\neq i}x +b_j) \iff W_ix + b_i > W_{j\neq i}x+b_j$$
bringing everything to one side gives:
$$\implies (W_{i1}-W_{j1})x_1+(W_{i2}-W_{j2})x_2+(W_{i3}-W_{j3})x_3 +...+ (b_i -b_j) > 0$$
so this decision rule does correspond to a separating hyperplane
Edit: it's a hyperplane for fixed $$j$$, but the inequality must hold for all $$j$$, so I guess it must be the region that's in common/intersection of all of the hyperplanes.....?

It would be interesting to obtain an explicit expression and solve for the set $$x$$ that satisfies a decision rule where instead of $$x$$ being the layer before the output layer, it instead would be the input to a multi-layer neural network