is logistic regression stochastic like neural network?

Question

I have observed that neural network models (using Keras TensorFlow) can be very unstable (when my sample size is small) in the sense that if I were to train 999 NN models, there might only be 99 with good training accuracy. I imagine this is due to the stochastic nature of the initiation of weights in the NN; hence only some initiation was able to lead to a local minima. However, when I use logistic regression (specifically the statsmodels package in python), the trained model is fairly stable in the sense that no matter how many times I train it, the accuracy and recall etc are fairly constant.

My question is - is this a consequence of the difference in nature between logistic regression and NN (e.g. could it be because logistic regression does not need random initiation of weights?) or is this merely a consequence of the packages I am using? (e.g. perhaps statsmodels has defined constant starting state?)

My understanding is that a logistic regression could also be viewed as a single node NN so I am wondering why should it be any different.

Answer 1

So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a convex optimization problem.

• What is happening here, when I use squared loss in logistic regression setting?
• is cost function of logistic regression convex or not?

When the design matrix is full rank and the data do not exhibit separation, logistic regression is strongly convex with a unique, finite minimizer. This means that a suitable optimization method will be able to recover the same minimizer across repeated runs, because there's only one minimum. These threads develop this topic in more detail.

• How to deal with perfect separation in logistic regression?
• Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what?
• Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple global minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail.

• Cost function of neural network is non-convex?
• Why is the cost function of neural networks non-convex?
• Can we use MLE to estimate Neural Network weights?

Examples strongly convex neural networks arise from special cases. The simplest example of a strongly convex neural network is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically generalized linear models (logistic regression, OLS, etc.). In particular, logistic regression is a generalized linear model (glm) in the sense that the logit of the estimated probability response is a linear function of the parameters. See: Why is logistic regression a linear model?

Answer 2

There is a key difference between logistic regression and neural networks. Neural Networks have multiple local minima and thus it's inherently sensible to kick off your gradient descent multiple times from different initialisations, as well as to use stochastic gradient descent. You would expect to end up in different places depending on where you start.

The logistic regression cost function however can be shown to be convex, and thus even if you kick your gradient descent off from different initialisations, you should always end up in the same place, give or take numerical effects associated with (S)GD.

It is true that logistic regression is a single layer neural network, but in somewhat handwaving terms, the term which goes through the logistic function is linear in all model parameters (the decision boundary is linear in all model parameters). As soon as you add another layer, the term which goes through the logistic function is a non-linear function of some of the model parameters. This is what starts to make the cost function non-convex (I state vaguely without proof), and that's why even a two-layer neural network will end up in different places if you initialise different and logistic regression is the special case

Answer 3

The reason why logistic regression appears more "stable" than neural networks (I'm assuming you mean multilayer perceptrons) is because of the difference of nature. We can summarize these differences in terms of the decision boundary and flexibility.

Logistic regression models are linear models (see the CV thread Why is logistic regression a linear classifier?), and so their decision boundaries are relatively constrained. NN's are highly nonlinear models (assuming you're using some form of nonlinear activation) that are able to form much more complex decision boundaries. Needless to say that this also leads us to the conclusion that logistic regression is less flexible than NN's are.

Answer 4

My understanding is that a logistic regression could also be viewed as a single node NN so I am wondering why should it be any different

Let's say you wanted to do a logistic regression with 4 outputs and 16 inputs using a neural network in TensorFlow. It might look something like this:

import tensorflow as tf

tf.random.set_seed(1)

model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(4, input_shape=(16,)))
model.add(tf.keras.layers.Softmax())

Now to answer your question:

is logistic regression stochastic like neural network?

That all depends on the optimization method used to train your logistic regression classifier or neural network. I haven't used the statsmodels package, but in TensorFlow you need to choose your optimizer. There are a number of built-in optimizers you can choose from.

Moreover, if you are wondering why each time you train your neural network that you get a different outcome, it is generally good practice to keep the random seed fixed throughout your experiments. This can easily be done by setting tf.random.set_seed(1) or any other fixed number. This should return the same result each time you train your model (assuming that all other hyperparameters were kept the same).

Answer 5

If we desire to model distribution of a binary (bernoulli-distributed) random variable, conditioned on a random vector ${\bf x}_n\in\mathbb{R}^M$, we could assume that

$$ t_n \vert {\bf x}_n \sim \text{Bern}(f({\bf x}_n)) $$

For some function $f:\mathbb{R}^M\to[0,1]$.

In a logistic regression, we choose $f({\bf x})=\sigma({\bf w}^T{\bf x}$), whilst for a feed-forward neural network (FFNN), we choose $f$ to be some complicated nonlinear function of the form

$$ f({\bf x}) = \sigma\left({{\bf w}^{(L)}}^Th\left({{\bf w}^{(L-1)}}^Th(...)\right)\right) $$

Whereas the logistic regression leads to a simple iterative equation to find its minimum, which always leads to the same minimum for a fixed dataset, the FFNN is dependent on the number of layers, the choice of $h$ and the disired number of parameters. Hence, it can be much more complicated to train an FFNN.