what makes neural networks a nonlinear classification model?

Question

I'm trying to understand the mathematical meaning of non-linear classification models:

I've just read an article talking about neural nets being a non-linear classification model.

But I just realize that:

The first layer:

$h_1=x_1∗w_{x1h1}+x_2∗w_{x1h2}$

$h_2=x_1∗w_{x2h1}+x_2∗w_{x2h2}$

The subsequent layer

$y=b∗w_{by}+h_1∗w_{h1y}+h_2∗w_{h2y}$

Can be simplified to

$=b′+(x_1∗w_{x1h1}+x_2∗w_{x1h2})∗w_{h1y}+(x_1∗w_{x2h1}+x_2∗w_{x2h2})∗w_{h2y} $

$=b′+x_1(w_{h1y}∗w_{x1h1}+w_{x2h1}∗w_{h2y})+x_2(w_{h1y}∗w_{x1h1}+w_{x2h2}∗w_{h2y}) $

An two layer neural network Is just a simple linear regression

$=b^′+x_1∗W_1^′+x_2∗W_2^′$

This can be shown to any number of layers, since linear combination of any number of weights is again linear.

What really makes an neural net a non linear classification model?
How the activation function will impact the non linearity of the model?
Can you explain me?

Answer 1

I think you forget the activation function in nodes in neural network, which is non-linear and will make the whole model non-linear.

In your formula is not totally correct, where,

$$ h_1 \neq w_1x_1+w_2x_2 $$

but

$$ h_1 = \text{sigmoid}(w_1x_1+w_2x_2) $$

where sigmoid function like this, $\text{sigmoid}(x)=\frac 1 {1+e^{-x}}$

Let's use a numerical example to explain the impact of the sigmoid function, suppose you have $w_1x_1+w_2x_2=4$ then $\text{sigmoid}(4)=0.99$. On the other hand, suppose you have $w_1x_1+w_2x_2=4000$, $\text{sigmoid}(4000)=1$ and it is almost as same as $\text{sigmoid}(4)$, which is non-linear.

---

In addition, I think the slide 14 in this tutorial can show where you did wrong exactly. For $H_1$ please not the otuput is not -7.65, but $\text{sigmoid}(-7.65)$

Answer 2

You're correct that multiple linear layers can be equivalent to a single linear layer. As the other answers have said, a nonlinear activation function allows nonlinear classification. Saying that a classifier is nonlinear means that it has a nonlinear decision boundary. The decision boundary is a surface that separates the classes; the classifier will predict one class for all points on one side of the decision boundary, and another class for all points on the other side.

Let's consider a common situation: performing binary classification with a network containing multiple layers of nonlinear hidden units and an output unit with a sigmoidal activation function. $y$ gives the output, $h$ is a vector of activations for the last hidden layer, $w$ is a vector of their weights onto the output unit, and $b$ is the output unit's bias. The output is:

$$y = \sigma(hw + b)$$

where $\sigma$ is the logistic sigmoid function. Output is interpreted as the probability that the class is $1$. The predicted class $c$ is:

$$c = \left \{ \begin{array}{cl} 0 & y \le 0.5 \\ 1 & y > 0.5 \\ \end{array} \right . $$

Let's consider the classification rule with respect to the hidden unit activations. We can see that the hidden unit activations are projected onto a line $hW + b$. The rule for assigning a class is a function of $y$, which is monotonically related to the projection along the line. The classification rule is therefore equivalent to determining whether the projection along the line is less than or greater than some threshold (in this case, the threshold is given by the negative of the bias). This means that the decision boundary is a hyperplane that's orthogonal to the line, and intersects the line at a point corresponding to that threshold.

I said earlier that the decision boundary is nonlinear, but a hyperplane is the very definition of a linear boundary. But, we've been considering the boundary as a function of the hidden units just before the output. The hidden unit activations are a nonlinear function of the original inputs, due to the previous hidden layers and their nonlinear activation functions. One way to think about the network is that it maps the data nonlinearly into some feature space. The coordinates in this space are given by the activations of the last hidden units. The network then performs linear classification in this space (logistic regression, in this case). We can also think about the decision boundary as a function of the original inputs. This function will be nonlinear, as a consequence of the nonlinear mapping from inputs to hidden unit activations.

This blog post shows some nice figures and animations of this process.

Answer 3

The nonlinearity comes from the sigmoid activation function, 1/(1+e^x), where x is the linear combination of predictors and weights that you referenced in your question.

By the way, the bounds of this activation are zero and one because either the denominator gets so large that the fraction approaches zero, or e^x becomes so small that the fraction approaches 1/1.