1
$\begingroup$

I'm looking for an out of the box transformer model that I can use that can give me document vectors for a list of text. I've looked at some of the BERT like transformers from huggingface but am unsure how to adapt them for document vectors, not word vectors, without retraining them so that the last state is somehow the only important state. I'd rather not take an old school approach and just average overall word vectors in a document either. While I'm moderately proficient with NLP, I'm not on the frontier really with what is going on with transformers so pointing to open source code available would help.

$\endgroup$
  • $\begingroup$ You seem to have introduced a new tag transformers. Can you please write a tag wiki? $\endgroup$ – kjetil b halvorsen Apr 25 '20 at 22:25
  • $\begingroup$ How do you do that? $\endgroup$ – www3 Apr 27 '20 at 0:03
  • $\begingroup$ Click on the tag, opens a new page with tag info, close to the top you see Improve tag info, click on that ... $\endgroup$ – kjetil b halvorsen Apr 27 '20 at 0:06
1
$\begingroup$

Transformers require quadratic memory with the length of the text, using too long texts might result in memory issues. E.g., in Huggingface's implementation, most of the models do not accept sequences longer than 512 subwords.

Making pre-trained Transformers work efficiently for long texts is an active research area, you can have a look at a paper called DocBERT to have an idea of what people are trying. But it will take some time until there is a nicely packaged open-source solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.