1
$\begingroup$

In the Keras documentation, an example states that a 2-layer neural network with 32 input neurons and 16 output neurons combined with a softmax activation on the output layer is equivalent to logistic regression. I haven't seen a formulation of logistic regression involving neurons before and I am having trouble seeing how to show that they are equivalent mathematically. Can anyone explain or point me towards some resources that might help?

$\endgroup$
2
  • 2
    $\begingroup$ Draw it out. They are right that (assuming the right activation functions) a neural network with no hidden layers is a logistic regression. $\endgroup$
    – Dave
    May 30, 2021 at 17:15
  • $\begingroup$ What they call neurons are literally nodes/features/columns. Eg in your example 32 input neurons means that you have dataset with 32 explanatory variables/columns. And you are using this to predict 16 groups. (ie since you have 16 output neurons). Softmax is the generalization of the logit function when you have more than 2 groups. Do you now understand why this is logistic regression? $\endgroup$
    – Onyambu
    May 30, 2021 at 17:19

1 Answer 1

1
$\begingroup$

Here is a drawing of a two-layer neural network.

enter image description here

The blue, red, purple, and grey lines represet network weights, and the black line is a bias.

Assume the pink output neuron to have sigmoid activation function so it gives a predicted probability. Then the equation is:

$$ p = \text{sigmoid}\left( b + \omega_{blue}x_{blue} + \omega_{red}x_{red} + \omega_{purple}x_{purple} + \omega_{grey}x_{grey} \right)\\ \iff\\ \log\left( \dfrac{ p }{ 1 - p } \right) = b + \omega_{blue}x_{blue} + \omega_{red}x_{red} + \omega_{purple}x_{purple} + \omega_{grey}x_{grey} $$

The second equation is that of a logistic regression (save for some statistical technicalities).

Since there is no limit to how many input neurons there could be, and the sigmoid activation function could have been the inverse of any of the link functions from generalized linear models, the Keras claim is not as strong as it could be. Indeed, generalized linear models (including OLS linear regression) can be expressed as two-layer neural networks.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.