If each neuron in a neural network is basically a logistic regression function, why multi layer is better? I'm going thru the Cousera's DeepAI course (Week3 video 1 "Neural Networks Overview") and Andrew Ng is explaining how each layer in a neural network is just another logistic regression, but he doesn't explain how it makes thing more accurate.  
So in a 2 layer network, how does calculating logistic multiple times make it more accurate?
 A: One way to see the power of nonlinearity is to note universal approximation theorem.
Though it's not very significant in practice (it's about capabilities of single layer networks), it tells you that if you use (arbitrary long) sums of sigmoids you can in principle approximate any continuous function to any desired level. If you know Fourier theory or remember Weierstrass approximation theorem it shouldn't be surprising.
A: When there are hidden layers exist in the neural network, we are adding non-linear features. Please check my answer here to get some sense.
what makes neural networks a nonlinear classification model?
Specifically, a nested sigmoid function will be more "powerful" than a linear transformation of original features and one sigmoid function (logistic regression.)

Here is an numerical example to address OP's comments.
Suppose we have data frame $X$, it is a $10 \times 3$ matrix (10 data points, 3 features.). If we want to have $7$ hidden unites, then the weight matrix $W$ is a $3 \times 7$ matrix. The output for the hidden layer (output of matrix multiplication $X \times W$) is a $10 \times 7$ matrix, which for each data point, there are $7$ expended features.
A: In standard logistic regression we have 1 output in the final layer. However with a single hidden layer neural network, we can have multiple intermediate values each of which can be thought of as an output of a different logistic regression model i.e. we are not just performing the same logistic regression again and again. It is then not a large jump to think that it is possible that the combination of these has greater expressive capabilities than the standard logistic regression model (and also has been shown in practice and theory).
You also mention in the comments about how these nodes have different values in the same layer if they have the same inputs? This is because they should have different weights. Each node in a neural network takes $N$ inputs and produces a value $\displaystyle y_j = f\left(\sum_{i = 1}^N w_{ji} \cdot x_i + b_j\right)$ where $f$ is some chosen function, in our case the sigmoid, $w_{ji}$ are the weights, $x_i$ are the inputs, and $b_j$ is some bias. The weights are chosen by an optimisation algorithm to optimise our objective e.g. minimise classification error. Initialisation is very important for the gradient descent algorithms that are usually used to optimise the weights. See https://intoli.com/blog/neural-network-initialization/ where if all the weights start off at 0, the network is unable to learn.
A: When using logistic activation functions, it's true that the function relating the inputs of each unit to its output is the same as for logistic regression. But, this isn't really the same as each unit performing logistic regression. The difference is that, in logistic regression, the weights and bias are chosen such that the output best matches given target values (using the log/cross-entropy loss). In contrast, hidden units in a neural net send their outputs to downstream units. There is no target output to match for individual hidden units. Rather, the weights and biases are chosen to minimize some objective function that depends on the final output of the network.
Rather than performing logistic regression, it might make more sense to think of each hidden unit as computing a coordinate in some feature space. From this perspective, the purpose of a hidden layer is to transform its input--the input vector is mapped to a vector of hidden layer activations. You can think of this as mapping the input into a feature space with a dimension corresponding to each hidden unit.
The output layer can often be thought of as a standard learning algorithm that operates in this feature space. For example, in a classification task, using a logistic output unit with cross entropy loss is equivalent to performing logistic regression in feature space (or multinomial logistic regression if using softmax outputs). In a regression task, using a linear output with squared error is equivalent to performing least squares linear regression in feature space.
Training the network amounts to learning the feature space mapping and classification/regression function (in feature space) that, together, give the best performance. Assuming nonlinear hidden units, increasing the width of the hidden layer or stacking multiple hidden layers permits more complex feature space mappings, thereby allowing more complex functions to be fit.
