1
$\begingroup$

As we know, the hidden layer unit has the following activation: $$h_t=tanh(UX_t+Wh_{t-1})$$ So there is the interaction between the input and the previous state: $UX_t+Wh_{t-1}$. My question is why it is an addition? What is the reason for not using $UX_t * Wh_{t-1}$?

$\endgroup$
1
$\begingroup$

The suggestion you state for multiplication would also be a viable approach. This does however have some downsides, since now there is a stronger interaction between the variables (think of positive and negative when multiplying). I suspect that the optimization is easier to perform for the most used approach due to simpler gradients and less interaction of weights.

$\endgroup$
1
$\begingroup$

Let's look at the problem in 1D

$\frac{\partial}{\partial x} g(f(x) + y) = g'(x + y) f'(x)$

Whereas

$\frac{\partial}{\partial x} g(f(x) y) = g'(f(x)y) f'(x)y$

As you see, the differential of this function does not only depend on value of $f(x)y$, it also depends on $y$ multiplicatively. Combined with the fact that in RNNs this rule is used many times this makes problems with exploding/vanishing gradient even worse if $y$ is not very close to 1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.