Skip to main content
2 of 4
content changes
wabbit
  • 370
  • 2
  • 14

Omitting the bias term, the recursion looks like: $$h_{t+1}=tanh(x_{t} U+W h_{t}) =tanh(p), \textrm{say}$$ where the tanh is taken elementwise.

Now, since $h_{t}$ and $h_{t+1}$ are vectors, the derivative $\frac{\partial h_{t+1}}{\partial h_{t}}$ is a Jacobian. Now,

  1. Notice that if $y=tanh(x), dy/dx=1-tanh^2(x)=1-y^2$

  2. Let's see how a single element of the Jacobian looks like. Assume that the hidden layers are of dimension $n$ Now $h_{t+1, i}=tanh(\sum_{k=1} ^{n} w_{ik}h_{t,k}+ g(x))$. Here $g(x)$ stands for some function of x.

  3. Hence $\frac{\partial h_{t+1, i}}{\partial h_{t, j}}$ = $1-tanh^2(h_{t+1,i})$

wabbit
  • 370
  • 2
  • 14