Is logistic regression a specific case of a neural network?

Question

I ended up in a debate regarding logistic regression and neural networks (NNs).

Is it wrong to say that logistic regression is a specific case of a neural network?

I have seen a lot of explanation in which logistic regression is shown as a NN, like the following:

From Tess Fernandez.

Or like this:

To me there are no differences, at least on the surface. There is a linear combination of the input, a fixed nonlinear function (sigmoid) and a classification based on the output probabilities, which is exactly a simple neural network with a single layer with a single node (at least in binary problem) and that use the sigmoid function as a nonlinear activation function.

But someone told me that it is not exactly so, because the assumptions behind this model are completely different from a neural network.

What are these assumptions? And why should logistic regression be considered different from a neural network?

I know that NN can handle more complex problems (like nonlinearly separable problems), but this puzzle me a bit.

Answer 1

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with no hidden layers and one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the neural-networks literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of bias. Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} -\sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a triplet-loss for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: What is happening here, when I use squared loss in logistic regression setting?).

Answer 2

Architecture-wise, yes, it's a special case of neural net. A logistic regression model can be constructed via neural network libraries. In the end, both have neurons having the same computations if the same activation and loss is chosen. This makes it a special NN, but since logistic regression is the simplest model, it's possible to train it using second-order methods, e.g. newton. Second order methods use Hessian matrix, in addition to gradients. But, this computation is not efficient for larger NNs and libraries prefer to use gradient descent alternatives or quasi-newton methods, that are either entirely first-order or approximate second-order methods. So, the slight difference lies in the possible optimisers, though this doesn't mean you'll get different solutions due to convexity properties of the problem (at least numerically).

Answer 3

If you have logistic activation function in the output layer and you are trying to maximise the log-likelihood of observations belonging to their corresponding classes (e.g. via its negative as the cost function), then yes, each output-layer neuron can be said to be an implementation of a logistic model over its inputs (which can be outputs of the neurons from the hidden layers).

In the simplest case, when you have a "network" of only one neuron with logistic activation function, and assuming you maximise the log-likelihood, then you are performing logistic regression.