Return to Answer

So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a convex optimization problem.

• The Effect of Using the MSE Score (Brier Score) for Logistic Regression
• is cost function of logistic regression convex or not?

When the design matrix is full rank and the data do not exhibit separation, logistic regression is strongly convex with a unique, finite minimizer. This means that a suitable optimization method will be able to recover the same minimizer across repeated runts. These threads develop this topic in more detail.

• How to deal with perfect separation in logistic regression?
• Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what?
• Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple global minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail.

• Cost function of neural network is non-convex?
• Why is the cost function of neural networks non-convex?
• Can we use MLE to estimate Neural Network weights?

Examples strongly convex neural networks arise from special cases. The simplest example of a strongly convex neural network is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically generalized linear models (logistic regression, OLS, etc.). In particular, logistic regression is a generalized linear model (glm) in the sense that the logit of the estimated probability response is a linear function of the parameters. See: Why is logistic regression a linear model?

So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a convex optimization problem.

• The Effect of Using the MSE Score (Brier Score) for Logistic Regression
• is cost function of logistic regression convex or not?

When the design matrix is full rank and the data do not exhibit separation, logistic regression is strongly convex with a unique, finite minimizer. This means that a suitable optimization method will be able to recover the same minimizer across repeated runs, because there's only one minimum. These threads develop this topic in more detail.

• How to deal with perfect separation in logistic regression?
• Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what?
• Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple global minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail.

• Cost function of neural network is non-convex?
• Why is the cost function of neural networks non-convex?
• Can we use MLE to estimate Neural Network weights?

Examples strongly convex neural networks arise from special cases. The simplest example of a strongly convex neural network is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically generalized linear models (logistic regression, OLS, etc.). In particular, logistic regression is a generalized linear model (glm) in the sense that the logit of the estimated probability response is a linear function of the parameters. See: Why is logistic regression a linear model?

So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a convex optimization problem.

• The Effect of Using the MSE Score (Brier Score) for Logistic Regression
• is cost function of logistic regression convex or not?

When the design matrix is full rank and the data do not exhibit separation, logistic regression is strongly convex with a unique, finite minimizer. This means that a suitable optimization method will be able to recover the same minimizer across repeated runts. These threads develop this topic in more detail.

• How to deal with perfect separation in logistic regression?
• Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what?
• Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple global minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail.

• Cost function of neural network is non-convex?
• Why is the cost function of neural networks non-convex?
• Can we use MLE to estimate Neural Network weights?

The only neural networks which are strongly convex are the ones where the parameters (weights, biases) are uniquely identified. The simplest examples of this is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically generalized linear models (logistic regression, OLS, etc.). In particular, logistic regression is a generalized linear model (glm) in the sense that the logit of the estimated probability response is a linear function of the parameters. See: Why is logistic regression a linear model?

So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a convex optimization problem.

• The Effect of Using the MSE Score (Brier Score) for Logistic Regression
• is cost function of logistic regression convex or not?

When the design matrix is full rank and the data do not exhibit separation, logistic regression is strongly convex with a unique, finite minimizer. This means that a suitable optimization method will be able to recover the same minimizer across repeated runts. These threads develop this topic in more detail.

• How to deal with perfect separation in logistic regression?
• Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what?
• Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple global minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail.

• Cost function of neural network is non-convex?
• Why is the cost function of neural networks non-convex?
• Can we use MLE to estimate Neural Network weights?

Examples strongly convex neural networks arise from special cases. The simplest example of a strongly convex neural network is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically generalized linear models (logistic regression, OLS, etc.). In particular, logistic regression is a generalized linear model (glm) in the sense that the logit of the estimated probability response is a linear function of the parameters. See: Why is logistic regression a linear model?

So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a convex optimization problem.

• The Effect of Using the MSE Score (Brier Score) for Logistic Regression
• is cost function of logistic regression convex or not?

When the design matrix is full rank and the data do not exhibit separation, logistic regression is strongly convex with a unique, finite minimizer. This means that a suitable optimization method will be able to recover the same minimizer across repeated runts. These threads develop this topic in more detail.

• How to deal with perfect separation in logistic regression?
• Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what?
• Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple global minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail.

• Cost function of neural network is non-convex?
• Why is the cost function of neural networks non-convex?
• Can we use MLE to estimate Neural Network weights?

The only neural networks which are strongly convex are the ones where the parameters (weights, biases) are uniquely identified. The simplest examples of this is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically linear models (logistic regression, OLS, etc.). In particular, logistic regression is a linear model in the sense that the logit of the estimated probability response is a linear function of the parameters. See: Why is logistic regression a linear model?

So far, no answer has addressed the core conceptual difference between logistic regression and neural networks.

Logistic regression is a convex optimization problem.

• The Effect of Using the MSE Score (Brier Score) for Logistic Regression
• is cost function of logistic regression convex or not?

When the design matrix is full rank and the data do not exhibit separation, logistic regression is strongly convex with a unique, finite minimizer. This means that a suitable optimization method will be able to recover the same minimizer across repeated runts. These threads develop this topic in more detail.

• How to deal with perfect separation in logistic regression?
• Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what?
• Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In general, neural networks are not a convex minimization problem. A core feature of a non-convex problem is that it has more than one minimum, possibly even multiple global minima. Multiple minima imply that a minimization scheme is susceptible to finding different solutions across different runs, especially when there is random component (random initialization, mini-batched training) to the optimization procedure. These threads develop this topic in more detail.

• Cost function of neural network is non-convex?
• Why is the cost function of neural networks non-convex?
• Can we use MLE to estimate Neural Network weights?

The only neural networks which are strongly convex are the ones where the parameters (weights, biases) are uniquely identified. The simplest examples of this is the neural network with no hidden layers and a monotonic activation for the output of the single linear output layer. These networks are identically generalized linear models (logistic regression, OLS, etc.). In particular, logistic regression is a generalized linear model (glm) in the sense that the logit of the estimated probability response is a linear function of the parameters. See: Why is logistic regression a linear model?