Variable length memory / information flow in Transformers My understanding is that RNNs, LTSMs and GRUNs can theoretically "remember" and "use" information in an input sequence spanning arbitrarily long distances, and one does not need to specify in any way the max. separation or distance between symbols in the input sequence that we may want the network to consider.
Do transformers (paper) have the same ability?
From my high-level understanding of transformers, they don't seem to have any recurrent information flow that would allow them to consider arbitrarily old inputs or outputs when decoding new inputs.
Or am I wrong? And if so, where in the following schematic from the original paper would the network capture that recurrent dependency? (i.e where in the circuit is information from an arbitrarily old past re-used?)

 A: RNN's operate on the input sequence one at a time going down the line.
Transformers have input width greater than the length of the longest input sequence. It eats up the whole sequence at once, chews it through the different attention layers, then spits it out. So it can attend to anywhere in the input at any time, but this means you can't run a given model on arbitrarily long input sequences the way you can with an RNN.
Here's a nice guided illustration of how the transformer computes its values:
http://jalammar.github.io/illustrated-transformer/
From the paper (bottom of page 6):

As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially
executed operations, whereas a recurrent layer requires $O(n)$ sequential operations. In terms of
computational complexity, self-attention layers are faster than recurrent layers when the sequence
length $n$ is smaller than the representation dimensionality $d$, which is most often the case with
sentence representations used by state-of-the-art models in machine translations, such as word-piece and byte-pair representations. To improve computational performance for tasks involving
very long sequences, self-attention could be restricted to considering only a neighborhood of size $r$ in the input sequence centered around the respective output position. This would increase the maximum
path length to $O(n/r)$. We plan to investigate this approach further in future work.

