I would like to understand how the block_size parameter in the huggingface transformers library works, particularly in comparison with model_max_length. I am interested in models being able to attend to long sequences.

It seems that long input sequences will be chopped into blocks according to the block size, so that each block fits into GPU memory. Therefore, the model won't see the whole sequence in finetuning and won't be able to attend to the whole sequence, only to the one block. If batch size matters here, let's assume it's small, even 1.

So, even if I use a long context model, the contex length is effectively determined by the block size. Doesn't matter if the model has 16k or 32k or 64k or 128k context length if the block size is smaller. Is this correct?

Also, how does it works at inference time? Is block_size baked into the model, or can I change it for prediction? Even if block_size doesn't come into play here, the model was finetuned with a given block_size, so it didn't learn how to process longer sequences (in terms of attending to the whole sequence). Or is it not true, and as long as the model can attend to the whole sequence it will be "fine", even though it wasn't trained for such long context?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.