How can calculate number of weights in LSTM

Question

If I have a LSTM with below parameters, how can I calculate the total number of wights ?

Input 39
Output 34
Hidden Layers = 3
Cells in each layer = 1024

I saw the Grave Presentation. In this file, in page 7 there is a sample of LSTM with these parameters:

Input = 205
Output = 205
Hidden Layers = 5
Cells in each layer = 700

unfortunately, I can't understand why this network has 21M weights?

Thank you.

Answer 1

Each cell in the LSTM has four components: the cell weights, the input gate, the forget gate, and the output gate. Each component has weights associated with all of its input from the previous layer, plus input from the previous time step. So if there are $n_i$ cells in an LSTM layer, and $n_{i-1}$ in the earlier layer, there will be $n_{i-1} + n_i$ inputs to each component of the cell. Since there are four components, that means there are $4(n_{i-1} + n_i)$ weights associated with each cell. And since we have $n_i$ cells, that means there are $4 n_i (n_{i-1} + n_i)$ weights associated with that layer.

In your first example, we have $n_0 = 39$, $n_1 = n_2 = n_3 = 1024$, and $n_4 = 34$. So the overall number of weights is about 21M.

In the second example we have $n_0 = 205$, $n_1 = ... = n_5 = 700$, and $n_6 = 205$. So the total number of weights is about 19M.

Answer 2

I think this may be my answer.

If LSTM don't use recurrent projection layer and non-recurrent projection layer, use this equivalent.

(nc*nc*4*di)+(ni*nc*4*di)+(nc*no)+(nc*3*di)

nc = number of LSTM cells
ni = number of input
no = numbre of output
di = number of layers

if LSTM use projection layer, number of weights obtain from this:

(nc*nr*4*di)+(ni*nc*4*di)+((nr+np)*no)+(nc*nr*di)+(nc*3*di)+(nr*nc*4*(di-1))

nr = number of recurrent projection layer
np = number of non-recurrent projection layer

For my exmaple : I use recurrent layer too. Recurrent projection layer = 256 Non-recurrent projection layer = 256

(1024*256*4*3)+(39*1024*4*3)+((256+256)*34)+(1024*256*3)+(1024*3*3)+(256*1025*4*(2)) = 6535168 = 6.5M

I used this reference: LONG SHORT-TERM MEMORY BASED RECURRENT NEURAL NETWORK ARCHITECTURES FOR LARGE VOCABULARY SPEECH RECOGNITION

Answer 3

Total number of weights in LSTM N/W = 4 x inp_dim x (inp_dim + out_dim + 1)

So, in your first model:

For Stage-1(input --> h1):

inp_dim = 39; out_dim = 1024 Therefore, weights of stage-1 = 4 x 39 x (39 + 1024 + 1) = 0.165M

For Stage-2(h1 --> h2):

inp_dim = 1024; out_dim = 1024 Therefore, weights of stage-2 = 4 x 1024 x (1024 + 1024 + 1) = 8.392M

For Stage-3(h2 --> h3):

inp_dim = 1024; out_dim = 1024 Therefore, weights of stage-3 = 4 x 1024 x (1024 + 1024 + 1) = 8.392M

For Stage-4(h3 --> output):

inp_dim = 1024; out_dim = 34 Therefore, weights of stage-4 = 4 x 1024 x (1024 + 34 + 1) = 4.337M

Thus, total weights = 0.165M + 8.392M + 8.392M + 4.337M = 21.2M (approx)

And, in your second model:

For Stage-1(input --> h1):

inp_dim = 205; out_dim = 700 Therefore, weights of stage-1 = 4 x 205 x (205 + 700 + 1) = 0.742M

For Stage-2(h1 --> h2):

inp_dim = 700; out_dim = 700 Therefore, weights of stage-2 = 4 x 700 x (700 + 700 + 1) = 3.922M

For Stage-3(h2 --> h3):

inp_dim = 700; out_dim = 700 Therefore, weights of stage-3 = 4 x 700 x (700 + 700 + 1) = 3.922M

For Stage-4(h3 --> h4):

inp_dim = 700; out_dim = 700 Therefore, weights of stage-4 = 4 x 700 x (700 + 700 + 1) = 3.922M

For Stage-5(h4 --> h5):

inp_dim = 700; out_dim = 700 Therefore, weights of stage-5 = 4 x 700 x (700 + 700 + 1) = 3.922M

For Stage-6(h5 --> output):

inp_dim = 700; out_dim = 205 Therefore, weights of stage-6 = 4 x 700 x (700 + 205 + 1) = 2.536M

Thus, total weights = 0.742M + (4 x 3.922M) + 2.536M = 19M (approx)