4
$\begingroup$

I was considering a feedforward neural network, where the output can be written as:

$y_o = \sigma(z_h)$,

where $z_h$ is the logit from the hidden layer, say, $w^T_{h}. x_h$, where the $h$ subscript denotes the hidden layer.

My question is since the hidden layer inputs $x_h$ are just the outputs from the input layer, can one just not write this as:

$y_o = \sigma(z_h) = \sigma(w^T_{h} . x_h) = \sigma(w^T_{h} y_i) = \sigma(w^T_{h} \sigma(w^T_{i} x_{i}))$, where this last equality explicitly shows the composition of functions (i denotes the input layer), and you don't have any $x_h$ terms in there?

Thanks.

$\endgroup$
3
$\begingroup$

Assuming you are using the same activation function, sigma, for both the hidden layer and output then yes, you are correct in that equality you've written. As for omitting the hidden layer nodes, if you physically omit them from the network you've simply got a modified linear regression with a different (sigma) final activation function.

$\endgroup$
  • $\begingroup$ Hi. Thanks for your answer. No, I didn't want to omit the hidden layer nodes, as the $w_h^T$ term is still present in the last equality. $\endgroup$ – Thomas Moore Dec 25 '19 at 5:48
  • 1
    $\begingroup$ Ah, I slightly misinterpreted what you were asking. The first sentence still holds though, you are correct in expanding it s.t. $x_h$ is replaced by $\sigma(w_i^Tx_i)$. $\endgroup$ – Jspang Dec 25 '19 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.