I'm looking for an out of the box transformer model that I can use that can give me document vectors for a list of text. I've looked at some of the BERT like transformers from huggingface but am unsure how to adapt them for document vectors, not word vectors, without retraining them so that the last state is somehow the only important state. I'd rather not take an old school approach and just average overall word vectors in a document either. While I'm moderately proficient with NLP, I'm not on the frontier really with what is going on with transformers so pointing to open source code available would help.
Transformers require quadratic memory with the length of the text, using too long texts might result in memory issues. E.g., in Huggingface's implementation, most of the models do not accept sequences longer than 512 subwords.
Making pre-trained Transformers work efficiently for long texts is an active research area, you can have a look at a paper called DocBERT to have an idea of what people are trying. But it will take some time until there is a nicely packaged open-source solution.