So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a **convex** optimization problem.
- https://stats.stackexchange.com/questions/326350/what-is-happening-here-when-i-use-squared-loss-in-logistic-regression-setting
- https://stats.stackexchange.com/questions/412650/is-cost-function-of-logistic-regression-convex-or-not

When the design matrix is full rank and the data do not exhibit separation, logistic regression is **strongly convex** with a unique, finite minimizer.  This means that a suitable optimization method will be able to recover the same minimizer across repeated runs, because there's only one minimum. These threads develop this topic in more detail.
- https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression
- https://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-perfect-separation-hauck-donner-phenomenon
- https://stats.stackexchange.com/questions/239928/is-there-any-intuitive-explanation-of-why-logistic-regression-will-not-work-for

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple *global* minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail. 
- https://stats.stackexchange.com/questions/106334/cost-function-of-neural-network-is-non-convex
- https://stats.stackexchange.com/questions/281240/why-is-the-cost-function-of-neural-networks-non-convex
- https://stats.stackexchange.com/questions/145902/can-we-use-mle-to-estimate-neural-network-weights/145907#145907

Examples strongly convex neural networks arise from special cases. The simplest example of a strongly convex neural network is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically *generalized* linear models (logistic regression, OLS, etc.). In particular, logistic regression is a *generalized linear model* ([tag:glm]) in the sense that the logit of the estimated probability response is a linear function of the parameters. See: https://stats.stackexchange.com/questions/88603/why-is-logistic-regression-a-linear-model