As we know, the hidden layer unit has the following activation: $$h_t=tanh(UX_t+Wh_{t-1})$$ So there is the interaction between the input and the previous state: $UX_t+Wh_{t-1}$. My question is why it is an addition? What is the reason for not using $UX_t * Wh_{t-1}$?
2 Answers
The suggestion you state for multiplication would also be a viable approach. This does however have some downsides, since now there is a stronger interaction between the variables (think of positive and negative when multiplying). I suspect that the optimization is easier to perform for the most used approach due to simpler gradients and less interaction of weights.
Let's look at the problem in 1D
$\frac{\partial}{\partial x} g(f(x) + y) = g'(x + y) f'(x)$
Whereas
$\frac{\partial}{\partial x} g(f(x) y) = g'(f(x)y) f'(x)y$
As you see, the differential of this function does not only depend on value of $f(x)y$, it also depends on $y$ multiplicatively. Combined with the fact that in RNNs this rule is used many times this makes problems with exploding/vanishing gradient even worse if $y$ is not very close to 1.