I'm looking for an out of the box transformer model that I can use that can give me document vectors for a list of text. I've looked at some of the BERT like transformers from huggingface but am unsure how to adapt them for document vectors, not word vectors, without retraining them so that the last state is somehow the only important state. I'd rather not take an old school approach and just average overall word vectors in a document either. While I'm moderately proficient with NLP, I'm not on the frontier really with what is going on with transformers so pointing to open source code available would help.
3
-
$\begingroup$ You seem to have introduced a new tag transformers. Can you please write a tag wiki? $\endgroup$– kjetil b halvorsen ♦Commented Apr 25, 2020 at 22:25
-
$\begingroup$ How do you do that? $\endgroup$– www3Commented Apr 27, 2020 at 0:03
-
$\begingroup$ Click on the tag, opens a new page with tag info, close to the top you see Improve tag info, click on that ... $\endgroup$– kjetil b halvorsen ♦Commented Apr 27, 2020 at 0:06
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Transformers require quadratic memory with the length of the text, using too long texts might result in memory issues. E.g., in Huggingface's implementation, most of the models do not accept sequences longer than 512 subwords.
Making pre-trained Transformers work efficiently for long texts is an active research area, you can have a look at a paper called DocBERT to have an idea of what people are trying. But it will take some time until there is a nicely packaged open-source solution.
$\begingroup$
$\endgroup$
Reformer was specifically designed to address memory issues. One of the authors of Reformer is Lukasz Kaiser, who was one of the authors of the transformer architecture.
In particular, Reformer solves the quadratic memory problem for attention.
An open source implementation is available in Trax
