2
$\begingroup$

In trying to get some sort of intuition into what goes on inside an LSTM, there is a step that has the potential to make things fall in place nicely.

Here is a diagram from this tutorial illustrating the step in question with nomenclature notes added (feel free to correct mistakes):

enter image description here

I understand $[h_{t-1}, x_t]$ as the vector (or matrix (?)) concatenating the previous activation layer or output $(h_{t-1})$ and the new input $x_t$, while sigma $(\sigma)$ stands for sigmoid activation. $\tilde C_t$ is the update gate (?) or proposed new cell state.

I presume that $\otimes$ in the diagram (minor design differences aside) is the Hadamard product.

Both $\sigma$ in the $i_t$ equation, and $\tanh$ in the $\tilde C_t$ formula take in the concatenated vector $[h_{t-1},x_t]$ with learned weights and biases.

The question is:

What is the intuition of the purpose behind obtaining the sigmoid activation and the tanh activation of the same vector, and then combining them through the element-wise multiplication (Hadamard)?

I see that sigmoid provides a weight for each element of the vector, but what is the purpose / convenience / usefulness in using $\tanh$? In part the answer lies in minimizing the vanishing gradient problem, but there are more patterns and ideas behind the $\sigma / \tanh$ interplay.

For instance, in this Quota answer:

When the portion of signal arrives, the gate regulates which parts of the signal should be allowed into the unit and how much of those parts should be allowed. That’s why the gates use sigmoid activation function which takes values from 0 to 1 and not tanh (-1 to 1), because it functions like a filter.

... this concepts seem to fall in place.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$
  • sigmoid squashes into the range 0 .. 1. The output of a sigmoid is like a binary on/off. But a binary on/off is not differentiable, cannot learn through back-propagation, so we 'soften' the binary on/off 0/1, to a sigmoid, which is differentiable
  • the tanh is the actual updates that we want to apply, everywhere that the sigmoid is outputting 1, or near to 1. tanh squashes into the range -1 .. 1. It can be negative as well as positive, since we might wish to apply negative changes to the cell state.
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you, Hugh, for your answer. Sensing a decreasing likelihood of responses, I continued digging out info, to the point of spilling over into a question + answer post. So I wonder if you could include some lines explaining that very specific operation vis-a-vis the difference between gate recurrent connections and cell recurrent connections. $\endgroup$
    – Antoni Parellada
    Apr 28, 2017 at 14:00
  • $\begingroup$ Please feel free to go ahead and put your new-found knowledge into a separate answer. Its one of the best ways to learn, and consolidate your knowledge :-) $\endgroup$
    – Hugh Perkins
    Apr 28, 2017 at 14:01
  • $\begingroup$ I have upvoted 3 excellent questions of yours to compensate for the un-accepted vote on this current answer. I believe the OP is open to a more extensive answer, perhaps spurred by my very tentative answer, and would like to "reopen" it. $\endgroup$
    – Antoni Parellada
    Jul 19, 2017 at 11:51
  • $\begingroup$ Can someone please provide a real example for the intuition behind tanh function? Before this there is a forget gate. Which could be considered as ignoring gender (mentioned in the blog). But for -1 rather than forgetting, are we deleting some information taken from the new input at this state t? As per my understanding, Sigmoid basically acts on previous state and tanh acts on current input. $\endgroup$
    – Naga Vemprala
    Jul 9, 2018 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.