Return to Answer

What you're describing is called a non-stationary RNN. At different time steps, you use different subsets of your parameters.

There's nothing terribly wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias.

The risks (which you may choose to accept!) are twofold.

1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$ and $W_{h}^{100}$. Its value will have higher variance.
2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.

Beyond that—what if during testing, you come across a sequence of length 101? You've never learned $W_{e}^{101}$ and $W_{h}^{101}$, so your model won't know what to do.

This is the benefit of sharing weights across time steps. You can use them to process any sequence length, even if unseen: 25, 101, or even 100000. While the last may be inadvisable, it's at least mathematically possible.

What you're describing is called a non-stationary RNN. At different time steps, you use different subsets of your parameters.

There's nothing terribly wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias.

The risks (which you may choose to accept!) are twofold.

1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$ and $W_{h}^{100}$. Its value will have higher variance.
2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.

Beyond that—what if during testing, you come across a sequence of length 101? You've never learned $W_{e}^{101}$ and $W_{h}^{101}$, so your model won't know what to do—the estimates here will be garbage.

This is the benefit of sharing weights across time steps. You can use them to process any sequence length, even if unseen: 25, 101, or even 100000. While the last may be inadvisable, it's at least mathematically possible.

What you're describing is called a non-stationary RNN. At different time steps, you use different subsets of your parameters.

There's nothing terribly wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias.

The risks (which you may choose to accept!) are twofold.

1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$. Its value will have higher variance.
2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.

Beyond that—what if during testing, you come across a sequence of length 101? You've never learned $W_{e}^{101}$, so your model won't know what to do.

This is the benefit of sharing weights across time steps. You can use them to process any sequence length, even if unseen: 25, 101, or even 100000. While the last may be inadvisable, it's at least mathematically possible.

What you're describing is called a non-stationary RNN. At different time steps, you use different subsets of your parameters.

There's nothing terribly wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias.

The risks (which you may choose to accept!) are twofold.

1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$ and $W_{h}^{100}$. Its value will have higher variance.
2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.

Beyond that—what if during testing, you come across a sequence of length 101? You've never learned $W_{e}^{101}$ and $W_{h}^{101}$, so your model won't know what to do.

This is the benefit of sharing weights across time steps. You can use them to process any sequence length, even if unseen: 25, 101, or even 100000. While the last may be inadvisable, it's at least mathematically possible.

What you're describing is called a non-stationary RNN. At different time steps, you use different subsets of your parameters.

There's nothing terribly wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias.

The risks (which you may choose to accept!) are twofold.

1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$. Its value will have higher variance.
2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.

What you're describing is called a non-stationary RNN. At different time steps, you use different subsets of your parameters.

There's nothing terribly wrong with this approach; you can definitely do it if you feel that the parameter-tying in your RNN is too restrictive of an inductive bias.

The risks (which you may choose to accept!) are twofold.

1. Parameters for positions farther into the sequence receive less information from training data. If the mean sequence length in your data is 25, but the max is 100, then very few examples will actually involve $W_{e}^{100}$. Its value will have higher variance.
2. Even if lengths are more consistent, introducing vastly more parameters will require more training data to achieve a comparable model fit. If you weren't underfitting, then you'll probably start overfitting by introducing these new parameters.

Beyond that—what if during testing, you come across a sequence of length 101? You've never learned $W_{e}^{101}$, so your model won't know what to do.

This is the benefit of sharing weights across time steps. You can use them to process any sequence length, even if unseen: 25, 101, or even 100000. While the last may be inadvisable, it's at least mathematically possible.