Why do RNNs share weight?

Question

If weights are not shared the number of parameters will be extremely large and difficult to compute which I understand.

I don't understand the argument that varying length inputs is taken care by sharing weights, as stated in many Cross Validated answers like Why are the weights of RNN/LSTM networks shared across time? or in this blog https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce. If in the architecture below, I use a different $W_{e}^{(t)}$ for each word at time $t$, then all of the $W_{e}^{(t)}$ will still have the same dimension because the embedding dimension for each word is same (every $e^{(t)}$ has same size). And if we similarly take different $W_{h}^{(t)}$ at every time step (assuming all hidden states have same number of nodes) then all $W_{h}^{(t)}$ will also have same dimensions. It will be equivalent to a series of vanilla NN's (inputs are embedding vector and previous hidden state vector of let's say dimesnion $e$ and $h$ respectively). Then the vanilla NN at $t$ time will output : $h^{(t)}=\sigma(W_{h}^{(t)}h^{(t-1)}+W_{e}^{(t)}e^{(t)}+bias)$
So how does using same $W_{h}$ and $W_{e}$ solve the problem of variable input sequene lengths?

Also, I know that in standard RNNs like below, the hidden state kind of stores the context from previous time steps. What is the interpretation of $W_{h}$ here?

Answer 1

What you're describing is called a non-stationary RNN. At different time steps, you use different subsets of your parameters.

There's nothing terribly wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias.

The risks (which you may choose to accept!) are twofold.

1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$ and $W_{h}^{100}$. Its value will have higher variance.
2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.

Beyond that—what if during testing, you come across a sequence of length 101? You've never learned $W_{e}^{101}$ and $W_{h}^{101}$, so your model won't know what to do—the estimates here will be garbage.

This is the benefit of sharing weights across time steps. You can use them to process any sequence length, even if unseen: 25, 101, or even 100000. While the last may be inadvisable, it's at least mathematically possible.