Skip to main content
Cross Validated

Return to Question

edited tags
Source Link
Arya McCarthy
Arya McCarthy
  • 9k
  • 1
  • 24
  • 58

If weights are not shared the number of parameters will be extremely large and difficult to compute which I understand. I

I don't understand the argument that varying length inputs is taken care by sharing weights, as stated in many stackexchangeCross Validated answers like Why are the weights of RNN/LSTM networks shared across time? or in this blog https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce  . If in the architecture below, I use a different $W_{e}^{(t)}$ for each word at time $t$, then all of the $W_{e}^{(t)}$ will still have the same dimension because the embedding dimension for each word is same (every $e^{(t)}$ has same size). And if we similarysimilarly take different $W_{h}^{(t)}$ at every time step (assuming all hidden states have same number of nodes) then all $W_{h}^{(t)}$ will also have same dimensions. It will be equivalent to a series of vanilla NN's (inputs are embedding vector and previous hidden state vector of let's say dimesnion $e$ and $h$ respectively). Then the vanilla NN at $t$ time will output : $h^{(t)}=\sigma(W_{h}^{(t)}h^{(t-1)}+W_{e}^{(t)}e^{(t)}+bias)$
So how does using same $W_{h}$ and $W_{e}$ solve the problem of variable input sequene lengths?

Also, I know that in standard RNNs like below, the hidden state kind of storestores the context from previous time steps so what. What is the interpretation of $W_{h}$ here?

enter image description here

If weights are not shared the number of parameters will be extremely large and difficult to compute which I understand. I don't understand the argument that varying length inputs is taken care by sharing weights as stated in many stackexchange answers like Why are the weights of RNN/LSTM networks shared across time? or in this blog https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce  . If in the architecture below I use different $W_{e}^{(t)}$ for each word at time $t$ then all of the $W_{e}^{(t)}$ will still have the same dimension because the embedding dimension for each word is same (every $e^{(t)}$ has same size). And if we similary take different $W_{h}^{(t)}$ at every time step (assuming all hidden states have same number of nodes) then all $W_{h}^{(t)}$ will also have same dimensions. It will be equivalent to a series of vanilla NN's (inputs are embedding vector and previous hidden state vector of let's say dimesnion $e$ and $h$ respectively). Then the vanilla NN at $t$ time will output : $h^{(t)}=\sigma(W_{h}^{(t)}h^{(t-1)}+W_{e}^{(t)}e^{(t)}+bias)$
So how does using same $W_{h}$ and $W_{e}$ solve the problem of variable input sequene lengths?

Also I know that in standard RNNs like below hidden state kind of store the context from previous time steps so what is the interpretation of $W_{h}$ here?

enter image description here

If weights are not shared the number of parameters will be extremely large and difficult to compute which I understand.

I don't understand the argument that varying length inputs is taken care by sharing weights, as stated in many Cross Validated answers like Why are the weights of RNN/LSTM networks shared across time? or in this blog https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce. If in the architecture below, I use a different $W_{e}^{(t)}$ for each word at time $t$, then all of the $W_{e}^{(t)}$ will still have the same dimension because the embedding dimension for each word is same (every $e^{(t)}$ has same size). And if we similarly take different $W_{h}^{(t)}$ at every time step (assuming all hidden states have same number of nodes) then all $W_{h}^{(t)}$ will also have same dimensions. It will be equivalent to a series of vanilla NN's (inputs are embedding vector and previous hidden state vector of let's say dimesnion $e$ and $h$ respectively). Then the vanilla NN at $t$ time will output : $h^{(t)}=\sigma(W_{h}^{(t)}h^{(t-1)}+W_{e}^{(t)}e^{(t)}+bias)$
So how does using same $W_{h}$ and $W_{e}$ solve the problem of variable input sequene lengths?

Also, I know that in standard RNNs like below, the hidden state kind of stores the context from previous time steps. What is the interpretation of $W_{h}$ here?

enter image description here

Source Link
zombiecs
zombiecs
  • 65
  • 5

Why do RNNs share weight?

If weights are not shared the number of parameters will be extremely large and difficult to compute which I understand. I don't understand the argument that varying length inputs is taken care by sharing weights as stated in many stackexchange answers like Why are the weights of RNN/LSTM networks shared across time? or in this blog https://towardsdatascience.com/recurrent-neural-networks-d4642c9bc7ce . If in the architecture below I use different $W_{e}^{(t)}$ for each word at time $t$ then all of the $W_{e}^{(t)}$ will still have the same dimension because the embedding dimension for each word is same (every $e^{(t)}$ has same size). And if we similary take different $W_{h}^{(t)}$ at every time step (assuming all hidden states have same number of nodes) then all $W_{h}^{(t)}$ will also have same dimensions. It will be equivalent to a series of vanilla NN's (inputs are embedding vector and previous hidden state vector of let's say dimesnion $e$ and $h$ respectively). Then the vanilla NN at $t$ time will output : $h^{(t)}=\sigma(W_{h}^{(t)}h^{(t-1)}+W_{e}^{(t)}e^{(t)}+bias)$
So how does using same $W_{h}$ and $W_{e}$ solve the problem of variable input sequene lengths?

Also I know that in standard RNNs like below hidden state kind of store the context from previous time steps so what is the interpretation of $W_{h}$ here?

enter image description here