6
$\begingroup$

I was trying to figure out how to estimate the number of parameters in an LSTM layer. What is the relationship of number of parameters with the num lstm-cells, input-dimension, and hidden output-state dimension of the LSTM layer?

If the LSTM input is 512-d (word embedding dimension), output hidden dimension is 256, and there are 256 lstm units (bidirectional layer) in each of the bidirectional LSTM layers, what's the params per cell and total in the layers?

I came across this link https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network, and it seems to suggest that hidden output state dimension = number of lstm cells in the layer. Why is that?

$\endgroup$
20
  • 1
    $\begingroup$ Each cell has its own hidden state. $\endgroup$ May 27, 2020 at 18:21
  • 1
    $\begingroup$ @SycoraxsaysReinstateMonica I'm not sure I follow. what does that have to do with the question in the OP? $\endgroup$
    – Joe Black
    May 27, 2020 at 18:26
  • $\begingroup$ You write "[...] it seems to suggest that hidden output state dimension = number of lstm cells in the layer. Why is that?" And I wrote "Each cell has its own hidden state." If each of $n$ cells has its own hidden state, the dimension is $n$ when you collect those states together. $\endgroup$ May 27, 2020 at 18:28
  • 1
    $\begingroup$ @SycoraxsaysReinstateMonica but each cell's hidden state is 256-dimensional, i.e. it has 256 floating-point numbers in it. So if you collected each cell's hidden state, it'd be 256*256, for 256 cells in the layer. It's not n but n*n. $\endgroup$
    – Joe Black
    May 27, 2020 at 18:33
  • 1
    $\begingroup$ I don't think that's correct. Each cell's hidden state is 1 float. The reason you'd have output dimension 256 is because you have 256 units. Each unit produces 1 output dimension. For example, see pytorch.org/docs/stable/nn.html . If we look at the output, is has shape (num_layers * num_directions, batch, hidden_size). For1 layer, 1 direction, and batch size 1, we have hidden_size floats in total. $\endgroup$ May 27, 2020 at 18:47

2 Answers 2

3
$\begingroup$

I came across this link https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network, and it seems to suggest that hidden output state dimension = number of lstm cells in the layer. Why is that?

Each cell's hidden state is 1 float. As an example, the reason you'd have output dimension 256 is because you have 256 units. Each unit produces 1 output dimension.

For example, see this documentation page for Pytorch https://www.pytorch.org/docs/stable/nn.html. If we look at the output entry for an LSTM, the hidden state has shape (num_layers * num_directions, batch, hidden_size). So for a model with 1 layer, 1 direction (i.e. not bidirectional), and batch size 1, we have hidden_size floats in total.

You can also see this if you keep track of the dimensions used in the LSTM computation. At each timestep (element of the input sequence) the layer of an LSTM carries out these operations, which are just compositions of matrix-vector products and activation functions. $$ \begin{aligned} i_t &= \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{t-1} + b_{hi}) \\ f_t &= \sigma(W_{if} x_t + b_{if} + W_{hf} h_{t-1} + b_{hf}) \\ g_t &= \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{t-1} + b_{hg}) \\ o_t &= \sigma(W_{io} x_t + b_{io} + W_{ho} h_{t-1} + b_{ho}) \\ c_t &= f_t \odot c_{t-1} + i_t \odot g_t \\ h_t &= o_t \odot \tanh(c_t) \\ \end{aligned} $$ We're focused on the hidden state, $h_t$, so look at the operations involving $h_{t-1}$ because this is the hidden state at the previous time-step. The hidden-to-hidden connections must have have size hidden_size by hidden_size because they're matrices which must by conformable in a matrix-vector product where the vector has size hidden_size. The input-to-hidden connections must have size hidden size by input size because this is a matrix-vector product where the vector has size input size.

Importantly, your distinction between hidden size and number of units never makes an appearance. If hidden size and number of units were different, then this matrix-vector arithmetic would, somewhere, not be conformable because it wouldn't have compatible dimension.

As for counting the number of parameters in an LSTM model, see How can calculate number of weights in LSTM

I believe the confusion arises because OP has confused the hidden output state, which is an output of the model, with the weights of the hidden state. I think this is the case because you insist that the hidden state has shape (n,n). It’s not, but the hidden weights are square matrices. LSTM cells have memory, which is returned as a part of the output. This is used together with the model weights and biases to yield the prediction for the next time step. The difference between hidden state output and the hidden weights is that the model weights are the same for all time steps, while the hidden state can vary. This “memory” component is where LSTMs get their name.

$\endgroup$
3
  • $\begingroup$ i posted this in comment in op. i don't there is any reason for this assumption of 1 float produced by a lstm-cell. " Each cell's hidden state is 1 float. As an example, the reason you'd have output dimension 256 is because you have 256 units. Each unit produces 1 output dimension." i.stack.imgur.com/MLGXm.png shows h_t is of size hidden-dim (which is 256) not 1 float. $\endgroup$
    – Joe Black
    Jun 7, 2020 at 2:07
  • $\begingroup$ i don't think i'm actually conflating, and i think we are talking past each other. Looking at LSTM pytorch link, "h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len", isn't h_n here the tensor only for the last hidden-state in the sequence at time t, not for all hidden-states? $\endgroup$
    – Joe Black
    Jun 7, 2020 at 5:21
  • 1
    $\begingroup$ @JoeBlack yes, it’s the last time step. But that seems immaterial since we agree sequence length and hidden size are different. Are you sure you’re asking about the size of the hidden output and not asking about the number of parameters in the weights matrices? $\endgroup$ Jun 7, 2020 at 11:50
1
$\begingroup$

Regarding the question: "why the dimension of the hidden state is related to the number of cells in a LSTM layer"?, what I understand, a layer of 4 cells would be represented as the picture I attached.

It is clear with the picture that the state H has dimension 4, which is directly related to the number of cells (hidden states) of the layer. I hope that clarifies the original question, and please correct me if I'm wrong. LSTM layer with 4 cells

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.