Skip to main content
1 of 2

Why do RNNs share weight?

If weights are not shared the number of parameters will be extremely large and difficult to compute which I understand. I don't understand the argument that varying length inputs is taken care by sharing weights as stated in many stackexchange answers like Why are the weights of RNN/LSTM networks shared across time? or in this blog https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce . If in the architecture below I use different $W_{e}^{(t)}$ for each word at time $t$ then all of the $W_{e}^{(t)}$ will still have the same dimension because the embedding dimension for each word is same (every $e^{(t)}$ has same size). And if we similary take different $W_{h}^{(t)}$ at every time step (assuming all hidden states have same number of nodes) then all $W_{h}^{(t)}$ will also have same dimensions. It will be equivalent to a series of vanilla NN's (inputs are embedding vector and previous hidden state vector of let's say dimesnion $e$ and $h$ respectively). Then the vanilla NN at $t$ time will output : $h^{(t)}=\sigma(W_{h}^{(t)}h^{(t-1)}+W_{e}^{(t)}e^{(t)}+bias)$
So how does using same $W_{h}$ and $W_{e}$ solve the problem of variable input sequene lengths?

Also I know that in standard RNNs like below hidden state kind of store the context from previous time steps so what is the interpretation of $W_{h}$ here?

enter image description here