32
$\begingroup$

I ended up in a debate regarding logistic regression and neural networks (NNs).

Is it wrong to say that logistic regression is a specific case of a neural network?

I have seen a lot of explanation in which logistic regression is shown as a NN, like the following:

Enter image description here

From Tess Fernandez.

Or like this:

Enter image description here

To me there are no differences, at least on the surface. There is a linear combination of the input, a fixed nonlinear function (sigmoid) and a classification based on the output probabilities, which is exactly a simple neural network with a single layer with a single node (at least in binary problem) and that use the sigmoid function as a nonlinear activation function.

But someone told me that it is not exactly so, because the assumptions behind this model are completely different from a neural network.

What are these assumptions? And why should logistic regression be considered different from a neural network?

I know that NN can handle more complex problems (like nonlinearly separable problems), but this puzzle me a bit.

$\endgroup$
3
  • 3
    $\begingroup$ "because the assumptions behind this model are completely different from a neural network." I'd follow up on that. What are the different assumptions? It would not be so bizarre to try out a logistic regression before moving to more complex models like deep neural nets. $\endgroup$
    – Dave
    Dec 15, 2020 at 14:39
  • 1
    $\begingroup$ See stats.stackexchange.com/q/362425/60613 $\endgroup$
    – Firebug
    Dec 15, 2020 at 16:08
  • 2
    $\begingroup$ Two more comments worth making: 1) If your experience is with logistic regression but not with more complicated neural networks, don't advertise yourself as a neural net guru (even if you do have expertise in logistic regression) just because logistic regression is a particular case of a neural network. $\text{//}$ 2) The same logic about a logistic regression being a neural net with no hidden layers applies to a linear regression being a neural net regression with no hidden layers (so also don't advertise yourself as a neural net guru just because you know linear regression). $\endgroup$
    – Dave
    Dec 16, 2020 at 15:13

3 Answers 3

49
$\begingroup$

You have to be very specific about what you mean. We can show mathematically that a certain neural network architecture trained with a certain loss coincides exactly with logistic regression at the optimal parameters. Other neural networks will not.

A binary logistic regression makes predictions $\hat{y}$ using this equation: $$ \hat{y}=\sigma(X \beta + \beta_0) $$ where $X$ is a $n \times p$ matrix of features (predictors, independent variables) and vector $\beta$ is the vector of $p$ coefficients and $\beta_0$ is the intercept and $\sigma(z)=\frac{1}{\exp(-z)+1}$. Conventionally in a logistic regression, we would roll the $\beta_0$ scalar into the vector $\beta$ and append a column of 1s to $X$, but I've moved it out of $\beta$ for clarity of exposition.

A neural network with no hidden layers and one output neuron with a sigmoid activation makes predictions using the equation $$ \hat{y}=\sigma(X \beta + \beta_0) $$ with $\hat{y},\sigma,X, \beta, \beta_0$ as before. Clearly, the equation is exactly the same. In the literature, $\beta_0$ is usually called a "bias," even though it has nothing to do with the statistical concept of . Otherwise, the terminology is identical.

A logistic regression has the Bernoulli likelihood as its objective function, or, equivalently, the Bernoulli log-likelihood function. This objective function is maximized: $$ \arg\max_{\beta,\beta_0} \sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$ where $y \in \{0,1\}$. We can motivate this objective function from a Bernoulli probability model where the probability of success depends on $X$.

A neural network can, in principle, use any loss function we like. It might use the so-called "cross-entropy" function (even though the "cross-entropy" can motivate any number of loss functions; see How to construct a cross-entropy loss for general regression targets?), in which case the model minimizes this loss function: $$ \arg\min_{\beta,\beta_0} -\sum_i \left[ y_i \log(\hat{y_i}) + (1-y_i)\log(1-\hat{y_i})\right] $$

In both cases, these objective functions are strictly convex (concave) when certain conditions are met. Strict convexity implies that there is a single minimum and that this minimum is a global. Moreover, the objective functions are identical, since minimizing a strictly convex function $f$ is equivalent to maximizing $-f$. Therefore, these two models recover the same parameter estimates $\beta, \beta_0$. As long as the model attains the single optimum, it doesn't matter what optimizer is used, because there is only one optimum for these specific models.

However, a neural network is not required to optimize this specific loss function; for instance, a for this same model would likely recover different estimates $\beta,\beta_0$. And the MSE/least squares loss is not convex in this problem, so that neural network would differ from logistic regression as well (see: What is happening here, when I use squared loss in logistic regression setting?).

$\endgroup$
3
  • 4
    $\begingroup$ (+1) Well said! Via different optimisers, I didn't mean different solutions. $\endgroup$
    – gunes
    Dec 15, 2020 at 16:21
  • $\begingroup$ You might want to be just a little more clear about the actual answer to the question here. I think you are basically saying "yes, a logistic regression is a neural network with these specific details", but it's not very clear - gunes' answer makes that much more clear, even though your answer contains much more useful information as to why, and in what way it is a special case. $\endgroup$
    – Joe
    Dec 17, 2020 at 20:35
  • 2
    $\begingroup$ @Joe I think the opening paragraph is very explicit about the answer to the question. The details and nuance are an important part of the answer: some neural networks are logistic regression, and some neural networks are not. Can you help me understand what you've found unclear? $\endgroup$
    – Sycorax
    Dec 17, 2020 at 20:42
15
$\begingroup$

Architecture-wise, yes, it's a special case of neural net. A logistic regression model can be constructed via neural network libraries. In the end, both have neurons having the same computations if the same activation and loss is chosen. This makes it a special NN, but since logistic regression is the simplest model, it's possible to train it using second-order methods, e.g. newton. Second order methods use Hessian matrix, in addition to gradients. But, this computation is not efficient for larger NNs and libraries prefer to use gradient descent alternatives or quasi-newton methods, that are either entirely first-order or approximate second-order methods. So, the slight difference lies in the possible optimisers, though this doesn't mean you'll get different solutions due to convexity properties of the problem (at least numerically).

$\endgroup$
8
$\begingroup$

If you have logistic activation function in the output layer and you are trying to maximise the log-likelihood of observations belonging to their corresponding classes (e.g. via its negative as the cost function), then yes, each output-layer neuron can be said to be an implementation of a logistic model over its inputs (which can be outputs of the neurons from the hidden layers).

In the simplest case, when you have a "network" of only one neuron with logistic activation function, and assuming you maximise the log-likelihood, then you are performing logistic regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.