I keep seeing this online, on Quora and Machine Learning subreddits but I don't get it. Here's some basic math to show otherwise:

We use this equation for the cell state:

$c_t = f_t \odot c_t\__1 + i_t \odot g_t$

Now, we want to compute:

$\frac{\partial c_t}{\partial c_t\__1}$

Apparently, it's equal to the forget gate:

$\frac{\partial c_t}{\partial c_t\__1} = f_t$

I've seen this in numerous heavily upvoted Quora answers, Reddit comments, blog posts, etc.

First of all, how can this be so if we have the $i$ and $g$ gates, which are both dependent on the previous hidden state, which is in turn dependent on the previous cell state? The chain rule would extend for longer and we'd have more derivative terms in there.

Now, even if we were to ignore $i$ and $g$—because the derivative from that expression would just add onto the whole thing—there's still the issue of $f$ being functionally dependent on the previous hidden state; since $f$ is also dependent on the previous hidden state and the hidden state is dependent on the previous cell state, then:

$\frac{\partial c_t}{\partial c_t\__1} = \frac{\partial f_t}{\partial c_t\__1}$

Given we have some nonlinearities like $\text{tanh}$ and $\text{sigmoid}$ here, along with weighting, there's no way that this derivative will turn out being equal to $f$.

A couple places where this is said to be true (I can't remember the rest):

Is it because of truncated backprop? If so, how often do we use truncated backprop for it to be apparent/present in these separate sources I've found online?

  • 4
    $\begingroup$ If you're going to downvote then at least explain why. It helps everyone. $\endgroup$ – MCKapur Apr 13 '17 at 5:45
  • $\begingroup$ MCKapur asked this question also on reddit and on quora $\endgroup$ – Oren Milman Oct 7 '18 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.