I was thinking about RNNs and notice that the update equation for an RNN could be:

$$ h_t = tanh( W_{hx} x_t +b_x) + tanh( W_{hh} h_{t-1} + b_h ) $$ $$ o_t = W_{oh} h_t + b_h $$

but instead are:

$$ h_t = tanh( W_{hx} x_t + W_{hh} h_{t-1} + b_h ) $$ $$ o_t = W_{oh} h_t + b_h $$

Why is it that they are not the way as in the first equation?

The only way I found to justify it to myself is by thinking about them one term at a time, i.e. imagine I get one input and one previous state. How can I combine them to make the next hidden state? Obviously not by just element-wise adding them directly since they are different dimensions possibly (even if they were passed through a non-linearity). How about projecting them somehow and then combining them first and then we can decide where the non-linearity goes:

$$ \langle W_h , h \rangle + \langle W_x x \rangle $$

we could do:

$$ \theta( \langle W_h , h \rangle + \langle W_x x \rangle + b_h ) $$


$$ \theta( \langle W_h , h \rangle + b_h )+ \theta( \langle W_x x \rangle + b_x) $$

if we were to write out both expressions and maybe think of $\theta$ as a non-linearity like say a cubic or a quadratic functions (i.e. really bad approximations to Gaussians or sigmoids), we would then be in trouble with the separate application application of the non-linearity because it would obviously be missing cross terms between h and x like hx or hx^2 etc. This just seems that in terms of functions that depend on time, we decreased the expressivity of the network.

Is that true? Or is that not correct? After thinking about it a bit more it seems that after some number of execution of time steps some cross terms might be able to to be introduced but only between very old time steps and much recent ones? i.e. there seems to be a delay or a skipping going around. Not sure.

In short, why do we combine the affine transformations first and then apply the non-linearity?


I'm really not familiar with the equations you wrote down, but I know my backpropagation :)

It's as simple as this (in my eyes): a neuron computes its input by taking all its incoming connections and multiplying the weight of the incoming connection with the activation from the source of the connection. Regardless of where the connection is coming from!!

I most implementations of neural networks in programming languages, a neuron has an array of all outgoing and incoming connections. The neuron does not know where the connection is coming from, it only knows the source activation and the connection weight.

Additionally, think of the following: an activation of a neuron is anywhere between the value of 0 and 1 for most programming languages. But when we perform this equation:

$$ h_t = tanh( W_{hx} x_t +b_x) + tanh( W_{hh} h_{t-1} + b_h ) $$

We will get a value between 0 and 2. Why would we want that?

  • 1
    $\begingroup$ just curious, why did you mention backprop in the beginning of the question? Seemed tangential. As your last comment, that what my question is about! :p $\endgroup$ Apr 20 '17 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.