3
$\begingroup$

Here is the structure of RNN (one layer):

enter image description here

We know that Batch Normalization does not work for RNN. Suppose two samples $x^1,x^2,$ in each hidden layer, different sample may have different time depth (for $h^1_{T_1},\ h^2_{T_2},$ $T_1$ and $T_2$ may different). Thus for some large $T$ (deep in time dimension), there may be only one sample, which makes the statistical mean and variance unreasonable.

However I don't understand why some samples will stop at some time levels? Could you give an example?

I understand as that two sample sequence inputs: $(x^1_1,\cdots,x^1_T),(x^2_1,\cdots,x^2_T)$ have same length ($=T$), then the lengths of their hidden layers $\Big((h^1_1)^{(l)},\cdots, (h^1_T)^{(l)}\Big),\Big((h^2_1)^{(l)},\cdots, (h^2_T)^{(l)}\Big)$ should be same for each layer ($=T$). How could dynamic on length happens?

$\endgroup$
2
  • $\begingroup$ Are you asking why sequences can have different lengths? $\endgroup$
    – Sycorax
    Apr 6, 2021 at 14:06
  • $\begingroup$ @Sycorax yes, the reason for different samples having different lengths on time dimension. Pls see my update. $\endgroup$ Apr 6, 2021 at 14:25

1 Answer 1

1
$\begingroup$

Lots of real-world sequences have different lengths. A very common example occurs when using RNNs for language modeling. Words, sentences, paragraphs and documents are all variable-length sequences because there's no requirement that any two sentences, paragraphs or documents have the same length. For fully-connected networks, variable-length sequences can be a challenge because standard FCNs require fixed-length inputs.

But the nice thing about RNNs is that each element of the sequence is processed one at a time, so the model can naturally be adapted to variable-length sequences. Usually, this means that the sequences are processed all the way to the end, and then all time-steps larger than the length of the input are masked.

$\endgroup$
6
  • $\begingroup$ Lots of real-world sequences have different lengths do you mean the bottom inputs $(x^i_1,\cdots,x^i_T)$ usually have different lengths, but the consequent hidden layer $\Big((h^i_1)^{(l)},\cdots,(h^i_T)^{(l)}\Big)$ have the fixed lengths (same as the length of bottom input?) since it is pre-defined structure of network? $\endgroup$ Apr 6, 2021 at 15:07
  • $\begingroup$ The inputs $x^i$ can have different lengths. Sequence $x^1$ can have length $T_1$ and sequence $x^2$ can have length $T_2 \neq T_1$. Hidden sequences (which is not a synonym for hidden layer) can have different lengths also. $\endgroup$
    – Sycorax
    Apr 6, 2021 at 15:09
  • $\begingroup$ Hidden sequences (which is not a synonym for hidden layer) can have different lengths also actually this is the essential place I don't understand, usually the length is pre-defined as the structure of network. Could you explain why? Or could you give me an example? $\endgroup$ Apr 6, 2021 at 15:16
  • $\begingroup$ Suppose your model is one-step-ahead prediction. This means you use $x^i_t$ to predict $x^i_{t+1}$. The hidden sequence will have length $T_i$ (if you retain the whole sequence). On the other hand, the hidden sequence could just length 1 (if you only retain the last element). Which is the case depends on how you've coded the model. $\endgroup$
    – Sycorax
    Apr 6, 2021 at 15:18
  • $\begingroup$ get it. Could you explain more on the second paragraph? Especially for sequences are processed all the way to the end, and then all time-steps larger than the length of the input are masked. Do you mean we can manually cut the length of hidden sequence and make them no longer than the input's? And what's our goal? $\endgroup$ Apr 6, 2021 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.