LSTM : shape of tensors?

Question

I'm trying to understand LSTM, using for instance http://colah.github.io/posts/2015-08-Understanding-LSTMs/

I get the overall idea, I guess. But I'm not quite sure I get the maths.

I'll set a very simple problem : I have a sequence of numbers and want to predict next number.

So x_t is of shape (1), and as I understand, h_t will be the prediction, so it should also be of shape (1). (I'm just here ignoring batch size)

Now, the equation producing h_t, using * operation should then have two operands of the same size as the result; that is, C_t and o_t should also be of shape (1).

Following on that idea, equation producing C_t forces also shape of (1) for f_t, i_t and ~C_t.

So... everything reduced to scalar real numbers in that case ?

what am I getting wrong ? because this would just not be able to learn much, would it ?

Answer 1

The input shape for a RNN is typically 3 dimensional:

1. Number of samples
2. Number of timesteps
3. Input dimensions (features)

So as you say you start with a sequence of numbers, that's basically the timesteps. To successfully train a NN you need several of those sequences, that's number of samples. The input dimensions is the number of input for each timestep. For example if you try to categorize the expected weather as 'Good' or 'Bad' based on the temperature and the wind of the last 10 hours using hourly measures then your input shape is (None, 10, 2). Where None means you can feed as much data series as you have, but each data series consist of 10 times of a pair of temperature and wind.
Having this input shape the context will inherit the same shape therefore C_t, f_t, i_t, and h_t will all be pairs of temperature and wind.
Perhaps a missing point is that you can use multiple units, then the same thing happens multiple times. More units can learn more patterns or more complex patterns.
The output of LSTM is either h_t (shape (2)) or the entire list of h with shape (10, 2). You use the latter one when you stack LSTMs. Nevertheless after the LSTM layer you always need to add a dense layer to interpret the outcome of the units and to combine them into the desired output shape. For Good/Bad classification the output shape can be (1) therefore a dense layer can be used with 1 neuron. For this example a softmax activation (of the dense layer) will make sure the result is categorized e.g. 0 or 1 that should also represent Good/Bad classification in training data.

Answer 2

Your numbers are correct, but the things is (as Alex R. pointed out in comments), $h_t$ is the internal state and not necessarily the final output of your system.

Hence, if you believe a 1-d state vector won't be able to capture much of a state/history/context, then you can make it wider. Then for the output, you can employ an output network that takes the state $h_t$ and produces an output, e.g. the next number.

I like the figure in this page, which shows vector sizes in parentheses: https://embarc.org/embarc_mli/doc/build/html/MLI_kernels/comm_lstm.html

To summarize, below are the input and output tensor dimensions of the components of the LTSM box:

---

Assume input $x_t$ has dim n, and state $h_t$ has dim k.

(Ignoring batch, sequence, samples, etc. Assuming no peepholes into $C_{t-1}$).

There are 4 1-layer neural nets (actually no hidden layers, just a bunch of logistic regression units, more or less) that compute:

1. forget gate ($f_t$) (activation sigmoid)
2. input gate ($i_t$) (activation sigmoid)
3. cell state update ($\hat C_t$) (activation tanh)
4. output gate ($o_t$) (activation sigmoid)

All of these networks have:

• Input dim = (n+k) (concatenation of input $x_t$ and hidden state $h_t$)
• Output dim = k (dimension of the vectors $f_t, i_t, \hat C_t, o_t$)

Then, cell state $C_t$ and hidden state $h_t$ are computed as ($*$ is element-wise multiplication):

• $C_t = f_t * C_{t-1} + i_t * \hat C_t$ (all have dim k)

  new cell state = forget some of old cell state + add some of new cell state update

• $h_t = o_t * tanh(C_t)$ (all have dim k)

  choose some of cell state, after passing it thru tanh