Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In a LSTM block we have:
The hidden state is transmitted to the next time step. But, according to this image from Colah's Blog:
an LSTM block transmits to the next time step both the output and the cell state.
So, what is the hidden state ? Is the cell state? Is a concatenation of the output with the cell state? What is it?
From https://arxiv.org/abs/1603.03827 (~LSTM for text classification):
We use a variant of RNN called Long Short Term Memory (LSTM). For the $t^{th}$ word in the short-text, an LSTM takes as input $\mathbf{x}_t, \mathbf{h}_{t-1}, \mathbf{c}_{t-1}$ and produces the hidden state $\mathbf{h}_t$ as well as the memory cell (a.k.a. cell state) $\mathbf{c}_t$ based on the following formulas: \begin{align*} \mathbf{i}_t &= \sigma(W_i \mathbf{x}_t + U_i \mathbf{h}_{t-1} + \mathbf{b}_i) \\ \mathbf{f}_t &= \sigma(W_f \mathbf{x}_t + U_f \mathbf{h}_{t-1} + \mathbf{b}_f) \\ \tilde{\mathbf{c}}_t &= \text{tanh}(W_c \mathbf{x}_t + U_c \mathbf{h}_{t-1} + \mathbf{b}_c) \\ \mathbf{c}_t &= \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \\ \mathbf{o}_t &= \sigma(W_o \mathbf{x}_t + U_o \mathbf{h}_{t-1} + \mathbf{b}_o) \\ \mathbf{h}_t &= \mathbf{o}_t \odot \text{tanh}(\mathbf{c}_t) \end{align*}
where $W_j \in \mathbb{R}^{n \times m}, \; U_j \in \mathbb{R}^{n \times n}$ are weight matrices and $ \mathbf{b}_j \in \mathbb{R}^n$ are bias vectors, for $j \in \{i, f, c, o\}$.
The symbols $\sigma(\cdot)$ and tanh$(\cdot)$ refer to the element-wise sigmoid and hyperbolic tangent functions, and $\odot$ is the element-wise multiplication. $\mathbf{h}_0 = \mathbf{c}_0 = \mathbf{0}$.
