1
$\begingroup$

I am studying NLP via cs224n from Stanford.
I am reading this lecture note now.

When you refer to the 5th page, they want to derive the gradient with respect to W for RNN, to show the mathematical reasoning behind the vanishing gradient problem.

Here, I found something weird for me.

Here, I have $h_{j} = Wf(h_{j-1}) + W^{hx}x_{[t]}$ and I want to calculate $\frac{{\sigma}h_{j}}{{\sigma}h_{j-1}}$.

Because second term of right-hand side is independent to $h_{j-1}$, from my calculation, $\frac{{\sigma}h_{j}}{{\sigma}h_{j-1}} = W {\cdot}diag[f'(h_{j-1})].$

But they say $\frac{{\sigma}h_{j}}{{\sigma}h_{j-1}} = W^T {\cdot}diag[f'(h_{j-1})].$

Why do they transpose? Their definition of Jacobian is most widely used one and it doesn't match with their result.

Who is right?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.