4
$\begingroup$

In a LSTM block we have:

  • The input
  • The output
  • The hidden state

The hidden state is transmitted to the next time step. But, according to this image from Colah's Blog:

enter image description here

an LSTM block transmits to the next time step both the output and the cell state.

So, what is the hidden state ? Is the cell state? Is a concatenation of the output with the cell state? What is it?

$\endgroup$

1 Answer 1

3
$\begingroup$

From https://arxiv.org/abs/1603.03827 (~LSTM for text classification):

We use a variant of RNN called Long Short Term Memory (LSTM). For the $t^{th}$ word in the short-text, an LSTM takes as input $\mathbf{x}_t, \mathbf{h}_{t-1}, \mathbf{c}_{t-1}$ and produces the hidden state $\mathbf{h}_t$ as well as the memory cell (a.k.a. cell state) $\mathbf{c}_t$ based on the following formulas: \begin{align*} \mathbf{i}_t &= \sigma(W_i \mathbf{x}_t + U_i \mathbf{h}_{t-1} + \mathbf{b}_i) \\ \mathbf{f}_t &= \sigma(W_f \mathbf{x}_t + U_f \mathbf{h}_{t-1} + \mathbf{b}_f) \\ \tilde{\mathbf{c}}_t &= \text{tanh}(W_c \mathbf{x}_t + U_c \mathbf{h}_{t-1} + \mathbf{b}_c) \\ \mathbf{c}_t &= \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \\ \mathbf{o}_t &= \sigma(W_o \mathbf{x}_t + U_o \mathbf{h}_{t-1} + \mathbf{b}_o) \\ \mathbf{h}_t &= \mathbf{o}_t \odot \text{tanh}(\mathbf{c}_t) \end{align*}

where $W_j \in \mathbb{R}^{n \times m}, \; U_j \in \mathbb{R}^{n \times n}$ are weight matrices and $ \mathbf{b}_j \in \mathbb{R}^n$ are bias vectors, for $j \in \{i, f, c, o\}$.

The symbols $\sigma(\cdot)$ and tanh$(\cdot)$ refer to the element-wise sigmoid and hyperbolic tangent functions, and $\odot$ is the element-wise multiplication. $\mathbf{h}_0 = \mathbf{c}_0 = \mathbf{0}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.