If each neuron in a neural network is basically a logistic regression function, why multi layer is better?

I'm going thru the Cousera's DeepAI course (Week3 video 1 "Neural Networks Overview") and Andrew Ng is explaining how each layer in a neural network is just another logistic regression, but he doesn't explain how it makes thing more accurate.

So in a 2 layer network, how does calculating logistic multiple times make it more accurate?

• Often to fit a good single logistic regression model, you need to do significant feature engennering. The two layer nn is attempting to do some of that work for you. – Matthew Drury Aug 30 '17 at 14:14
• @msk, have you ever created new features based on your original features to improve your logistic regression? This is what the hidden layer(s) does. – Ricardo Cruz Sep 5 '17 at 21:56

When using logistic activation functions, it's true that the function relating the inputs of each unit to its output is the same as for logistic regression. But, this isn't really the same as each unit performing logistic regression. The difference is that, in logistic regression, the weights and bias are chosen such that the output best matches given target values (using the log/cross-entropy loss). In contrast, hidden units in a neural net send their outputs to downstream units. There is no target output to match for individual hidden units. Rather, the weights and biases are chosen to minimize some objective function that depends on the final output of the network.

Rather than performing logistic regression, it might make more sense to think of each hidden unit as computing a coordinate in some feature space. From this perspective, the purpose of a hidden layer is to transform its input--the input vector is mapped to a vector of hidden layer activations. You can think of this as mapping the input into a feature space with a dimension corresponding to each hidden unit.

The output layer can often be thought of as a standard learning algorithm that operates in this feature space. For example, in a classification task, using a logistic output unit with cross entropy loss is equivalent to performing logistic regression in feature space (or multinomial logistic regression if using softmax outputs). In a regression task, using a linear output with squared error is equivalent to performing least squares linear regression in feature space.

Training the network amounts to learning the feature space mapping and classification/regression function (in feature space) that, together, give the best performance. Assuming nonlinear hidden units, increasing the width of the hidden layer or stacking multiple hidden layers permits more complex feature space mappings, thereby allowing more complex functions to be fit.

One way to see the power of nonlinearity is to note universal approximation theorem.

Though it's not very significant in practice (it's about capabilities of single layer networks), it tells you that if you use (arbitrary long) sums of sigmoids you can in principle approximate any continuous function to any desired level. If you know Fourier theory or remember Weierstrass approximation theorem it shouldn't be surprising.

• This was the only person that actually answered the question properly, albeit a little too succinctly. A hidden layer is enough to create powerful new features that combine the original features. The problem is that you might need a hidden layer with too many nodes, and the convergence process works best with a deeper network with hidden layers > 1. – Ricardo Cruz Sep 5 '17 at 21:55

When there are hidden layers exist in the neural network, we are adding non-linear features. Please check my answer here to get some sense.

what makes neural networks a nonlinear classification model?

Specifically, a nested sigmoid function will be more "powerful" than a linear transformation of original features and one sigmoid function (logistic regression.)

Suppose we have data frame $X$, it is a $10 \times 3$ matrix (10 data points, 3 features.). If we want to have $7$ hidden unites, then the weight matrix $W$ is a $3 \times 7$ matrix. The output for the hidden layer (output of matrix multiplication $X \times W$) is a $10 \times 7$ matrix, which for each data point, there are $7$ expended features.
You also mention in the comments about how these nodes have different values in the same layer if they have the same inputs? This is because they should have different weights. Each node in a neural network takes $N$ inputs and produces a value $\displaystyle y_j = f\left(\sum_{i = 1}^N w_{ji} \cdot x_i + b_j\right)$ where $f$ is some chosen function, in our case the sigmoid, $w_{ji}$ are the weights, $x_i$ are the inputs, and $b_j$ is some bias. The weights are chosen by an optimisation algorithm to optimise our objective e.g. minimise classification error. Initialisation is very important for the gradient descent algorithms that are usually used to optimise the weights. See https://intoli.com/blog/neural-network-initialization/ where if all the weights start off at 0, the network is unable to learn.