In the Keras documentation, an example states that a 2-layer neural network with 32 input neurons and 16 output neurons combined with a softmax activation on the output layer is equivalent to logistic regression. I haven't seen a formulation of logistic regression involving neurons before and I am having trouble seeing how to show that they are equivalent mathematically. Can anyone explain or point me towards some resources that might help?
2
-
2$\begingroup$ Draw it out. They are right that (assuming the right activation functions) a neural network with no hidden layers is a logistic regression. $\endgroup$– DaveMay 30, 2021 at 17:15
-
$\begingroup$ What they call neurons are literally nodes/features/columns. Eg in your example 32 input neurons means that you have dataset with 32 explanatory variables/columns. And you are using this to predict 16 groups. (ie since you have 16 output neurons). Softmax is the generalization of the logit function when you have more than 2 groups. Do you now understand why this is logistic regression? $\endgroup$– OnyambuMay 30, 2021 at 17:19
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
Here is a drawing of a two-layer neural network.
The blue, red, purple, and grey lines represet network weights, and the black line is a bias.
Assume the pink output neuron to have sigmoid activation function so it gives a predicted probability. Then the equation is:
$$ p = \text{sigmoid}\left( b + \omega_{blue}x_{blue} + \omega_{red}x_{red} + \omega_{purple}x_{purple} + \omega_{grey}x_{grey} \right)\\ \iff\\ \log\left( \dfrac{ p }{ 1 - p } \right) = b + \omega_{blue}x_{blue} + \omega_{red}x_{red} + \omega_{purple}x_{purple} + \omega_{grey}x_{grey} $$
The second equation is that of a logistic regression (save for some statistical technicalities).
Since there is no limit to how many input neurons there could be, and the sigmoid activation function could have been the inverse of any of the link functions from generalized linear models, the Keras claim is not as strong as it could be. Indeed, generalized linear models (including OLS linear regression) can be expressed as two-layer neural networks.
