Return to Answer

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the neural-networks literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of bias. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} -\sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a triplet-loss for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: The Effect of Using the MSE Score (Brier Score) for Logistic Regression).

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with no hidden layers and one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the neural-networks literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of bias. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} -\sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a triplet-loss for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: The Effect of Using the MSE Score (Brier Score) for Logistic Regression).

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the neural-networks literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of bias. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} - \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a triplet-loss for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: The Effect of Using the MSE Score (Brier Score) for Logistic Regression).

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the neural-networks literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of bias. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} -\sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a triplet-loss for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: The Effect of Using the MSE Score (Brier Score) for Logistic Regression).

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the neural-networks literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of bias. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i}) $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} - \sum_i y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i}) $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a triplet-loss for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: The Effect of Using the MSE Score (Brier Score) for Logistic Regression).

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the neural-networks literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of bias. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} - \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a triplet-loss for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: The Effect of Using the MSE Score (Brier Score) for Logistic Regression).