0
$\begingroup$

Assume we have a neural network with 2 input neurons, 1 hidden neuron and 1 output neuron. The output of such a network is calculated as follows:

$$ f(w_1x_1 + w_2x_2)w_3$$ , where $f$ is the activation function of the hidden neuron, $x_1$ and $x_2$ are the network inputs, $w_1$ and $w_2$ the connection weights from the input neurons to the hidden neuron and $w_3$ the connection weight from the hidden neuron to the output neuron.

If $f$ is a linear activation function, the above neural network is identical to a linear model with coefficients $w_1w_3$ and $w_2w_3$.

Is it possible to derive the coefficients of a linear model if the activation function $f$ is a sigmoid or tanh function? If yes, how?

$\endgroup$
  • $\begingroup$ Is the sigmoid supposed to be the activation of the output or of the hidden neuron or both? $\endgroup$ – Sebastian Dec 22 '19 at 20:49
  • $\begingroup$ @Sebastian The output neuron is linear, only the hidden neuron sigmoid. $\endgroup$ – Julian Dec 23 '19 at 9:24
0
$\begingroup$

No, it's not possible (even with bias terms) because you won't be able to solve the coefficients $a,b$ in terms of $w_i$: $$f(w_1x_1+w_2x_2)w_3=ax_1+bx_2$$Intuitively, linear model has range $\mathbb R$, but if you use sigmoid, the range will be limited to $(w_3,-w_3)$. Only when used multiple hidden neurons, you can emulate the behaviour you want.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.