5
$\begingroup$

From hugginface documentation:

Transformer-based models are unable to process long sequences due to their self-attention operation

How long is "long" here? 1000 inputs? 10,000 inputs?

Just as a rough estimate, how long can the input sequence be before the transformer is impossible to train?

$\endgroup$
5
  • $\begingroup$ Depends how much time you have. Computing self-attention requires memory quadratic in the sequence length. $\endgroup$ Commented Apr 17, 2021 at 13:25
  • $\begingroup$ My point is that there’s a huge chasm between “impossible” and “impractical”. You’ll find libraries that cap the length at about 512. $\endgroup$ Commented Apr 17, 2021 at 13:36
  • $\begingroup$ ok maybe impractical was the correct word to use. So it should be between 0-200 length for optimality then? If there is a literal cap at 512 then the authors are suggesting that this is the max reasonable upper limit. $\endgroup$ Commented Apr 17, 2021 at 14:01
  • 1
    $\begingroup$ Remember that the practical limit depends on current hardware, the size of the data, and how long you’re willing to wait. But you can also set the limit in, e.g., huggingface, to whatever you want. $\endgroup$ Commented Apr 17, 2021 at 14:12
  • $\begingroup$ The Longformer uses a sliding window attention mechanism to allow for larger input sequences: huggingface.co/allenai/longformer-base-4096 $\endgroup$
    – slim
    Commented Jul 6, 2022 at 11:58

1 Answer 1

6
$\begingroup$

There is no theoretical limit on the input length (ie number of tokens for a sentence in NLP) for transformers. However in practice, longer inputs will consume more memory.

A slightly related question with more detailed answers: Why do attention models need to choose a maximum sentence length?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.