I have observed that neural network models (using Keras TensorFlow) can be very unstable (when my sample size is small) in the sense that if I were to train 999 NN models, there might only be 99 with good training accuracy. I imagine this is due to the stochastic nature of the initiation of weights in the NN; hence only some initiation was able to lead to a local minima. However, when I use logistic regression (specifically the
statsmodels package in python), the trained model is fairly stable in the sense that no matter how many times I train it, the accuracy and recall etc are fairly constant.
My question is - is this a consequence of the difference in nature between logistic regression and NN (e.g. could it be because logistic regression does not need random initiation of weights?) or is this merely a consequence of the packages I am using? (e.g. perhaps
statsmodels has defined constant starting state?)
My understanding is that a logistic regression could also be viewed as a single node NN so I am wondering why should it be any different.