How can I derive a linear model from a neural network with one hidden neuron?

Question

Assume we have a neural network with 2 input neurons, 1 hidden neuron and 1 output neuron. The output of such a network is calculated as follows:

$$ f(w_1x_1 + w_2x_2)w_3$$ , where $f$ is the activation function of the hidden neuron, $x_1$ and $x_2$ are the network inputs, $w_1$ and $w_2$ the connection weights from the input neurons to the hidden neuron and $w_3$ the connection weight from the hidden neuron to the output neuron.

If $f$ is a linear activation function, the above neural network is identical to a linear model with coefficients $w_1w_3$ and $w_2w_3$.

Is it possible to derive the coefficients of a linear model if the activation function $f$ is a sigmoid or tanh function? If yes, how?

Answer 1

No, it's not possible (even with bias terms) because you won't be able to solve the coefficients $a,b$ in terms of $w_i$: $$f(w_1x_1+w_2x_2)w_3=ax_1+bx_2$$Intuitively, linear model has range $\mathbb R$, but if you use sigmoid, the range will be limited to $(w_3,-w_3)$. Only when used multiple hidden neurons, you can emulate the behaviour you want.