In Recurrent NN, what's the reason for adding instead of multiplying the input term and the state term in the hidden units?

Question

As we know, the hidden layer unit has the following activation: $$h_t=tanh(UX_t+Wh_{t-1})$$ So there is the interaction between the input and the previous state: $UX_t+Wh_{t-1}$. My question is why it is an addition? What is the reason for not using $UX_t * Wh_{t-1}$?

Answer 1

The suggestion you state for multiplication would also be a viable approach. This does however have some downsides, since now there is a stronger interaction between the variables (think of positive and negative when multiplying). I suspect that the optimization is easier to perform for the most used approach due to simpler gradients and less interaction of weights.

Answer 2

Let's look at the problem in 1D

$\frac{\partial}{\partial x} g(f(x) + y) = g'(x + y) f'(x)$

Whereas

$\frac{\partial}{\partial x} g(f(x) y) = g'(f(x)y) f'(x)y$

As you see, the differential of this function does not only depend on value of $f(x)y$, it also depends on $y$ multiplicatively. Combined with the fact that in RNNs this rule is used many times this makes problems with exploding/vanishing gradient even worse if $y$ is not very close to 1.