I am studying NLP via cs224n from Stanford.
I am reading this lecture note now.

When you refer to the 5th page, they want to derive the gradient with respect to W for RNN, to show the mathematical reasoning behind the vanishing gradient problem.

Here, I found something weird for me.

Here, I have $h_{j} = Wf(h_{j-1}) + W^{hx}x_{[t]}$ and I want to calculate $\frac{{\sigma}h_{j}}{{\sigma}h_{j-1}}$.

Because second term of right-hand side is independent to $h_{j-1}$, from my calculation, $\frac{{\sigma}h_{j}}{{\sigma}h_{j-1}} = W {\cdot}diag[f'(h_{j-1})].$

But they say $\frac{{\sigma}h_{j}}{{\sigma}h_{j-1}} = W^T {\cdot}diag[f'(h_{j-1})].$

Why do they transpose? Their definition of Jacobian is most widely used one and it doesn't match with their result.

Who is right?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.