What you're describing is called a __non-stationary RNN__. At different time steps, you use different subsets of your parameters. There's nothing _terribly_ wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias. The risks (which you may choose to accept!) are twofold. 1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$. Its value will have higher variance. 2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.