1
$\begingroup$

I was reading a RNN paper that discuss vanishing/exploding gradient: http://proceedings.mlr.press/v28/pascanu13.pdf and when they present Eq. 2, they assume that $$ x_t = W_{rec} \sigma(x_{t-1}) + W_{in} u_t + b $$ And $$ x_t = \sigma(W_{rec} x_{t-1} + W_{in} u_t + b) $$ Are two equivalent formulations.

I tried to prove they were equivalent but wasn't able to. Someone has any hints regarding that? Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

Writing the first as $$ x_t = W_{rec} \sigma(x_{t-1}) + W_{in} u_t + b $$ and the second as $$ x'_t = \sigma(W_{rec} x'_{t-1} + W_{in} u_t + b) $$

you can see they are equivalent when $x'_t = \sigma(x_t)$. So any function $f$ you might want to compute on $x'_t$, you can also compute it by $f(x_t') = f(\sigma(x_t))$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.