I have a problem that is not particularly unique, but I'm still having trouble to figure out exactly how it's usually done.

My training set is of the form $\mathcal{T}=\{(t_i\in \mathbb{R}^{[n,m]\times 128} ,l_i\in \mathbb{B})\}_{i=1}^N$. So the input is variable length multivariate time series and the label is binary. I have some insight that self attention should be useful since the classification problem is related to the periodic behaviour of the input sequence.

This paper (RepNet) from CVPR 20 used a self-attention network (transformer) for analysis of a periodic signal with good results so my insight is coming mostly from here.

The problem I have is how to use a transformer network like torch.nn.TransformerEncoder for variable length input. This article explains that BERT models expect fixed length input so there is usually a padding character appended.

My main question is to find a method to do self attention on sequences of variable length, if it's possible.



2 Answers 2


There is nothing in the self-attention parameterization that would make it limited to a pre-defined length. The attention is done by a dot-product of all state-pairs and then as a weighted sum of the projected states.

The transformer encoder uses position encoding. This is the only component that could be length-dependent, however, this is not part of the TransformerEncoder class. You can either learn the position embeddings (that can only learn for positions in your training data) or use analytically computed (as can be seen in the PyTorch Transformers tutorial).

The only tricky part is that sequences in a single batch can have different lengths. You can avoid that by using a batch size of 1 (at the expense of a significant slowdown). Alternatively, you can provide the encoder with a binary mask telling what positions in the sequences are valid. It is done by the src_key_padding_mask attribute in the encoder call.

  • $\begingroup$ Thanks this is quite helpful $\endgroup$ Sep 24, 2020 at 17:07
  • 2
    $\begingroup$ Another solution for batching could be to use packed sequences and a custom collation function for the dataloader $\endgroup$ Sep 24, 2020 at 17:09

It would be a good idea to employ adaptive batching, that is you pad a batch(append zeros) by the longest case in that and in the reference mode the batch size is just one and the length is the length of that case. You don't need bucketing but enjoy the speeding up.

You can do that by first padding all cases in the data pipeline by the max length and second truncating any batch according to the longest actual length in the graph.

Since the time complexity is quadratic to the length of the input the reference would also be fast for those short inputs since the length is variable. For more information please refer to this tutorial: Smart Batching Tutorial - Speed Up BERT Training.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.