# How can I derive a linear model from a neural network with one hidden neuron?

Assume we have a neural network with 2 input neurons, 1 hidden neuron and 1 output neuron. The output of such a network is calculated as follows:

$$f(w_1x_1 + w_2x_2)w_3$$ , where $$f$$ is the activation function of the hidden neuron, $$x_1$$ and $$x_2$$ are the network inputs, $$w_1$$ and $$w_2$$ the connection weights from the input neurons to the hidden neuron and $$w_3$$ the connection weight from the hidden neuron to the output neuron.

If $$f$$ is a linear activation function, the above neural network is identical to a linear model with coefficients $$w_1w_3$$ and $$w_2w_3$$.

Is it possible to derive the coefficients of a linear model if the activation function $$f$$ is a sigmoid or tanh function? If yes, how?

• Is the sigmoid supposed to be the activation of the output or of the hidden neuron or both? Commented Dec 22, 2019 at 20:49
• @Sebastian The output neuron is linear, only the hidden neuron sigmoid. Commented Dec 23, 2019 at 9:24

No, it's not possible (even with bias terms) because you won't be able to solve the coefficients $$a,b$$ in terms of $$w_i$$: $$f(w_1x_1+w_2x_2)w_3=ax_1+bx_2$$Intuitively, linear model has range $$\mathbb R$$, but if you use sigmoid, the range will be limited to $$(w_3,-w_3)$$. Only when used multiple hidden neurons, you can emulate the behaviour you want.