Extracting word embedding features of a sentence using Transformer-XL

Question

As you know, there are several pre-trained models that we can use to extract word embeddings.

As an example, I can use the following codes to retrieve word2vec features of my text:

import gensim
from gensim.models import Word2Vec

word2vec_model = gensim.models.KeyedVectors.load_word2vec_format('./data/word_embeddings/GoogleNews-vectors-negative300.bin', binary=True)

def retrieve_word2vec(word):
    return word2vec_model.wv[word]

Currently, I am interested to extract word features using pre-trained transformer-XL models. Is there any sample code to understand how to do that? (I couldn't find something useful by myself).

Thanks in advance

Answer 1

I'm not sure if you're asking specifically about gensim, and this question will likely be closed (generally questions about specific libraries aren't accepted) but I really like the Huggingface library and they have a transformerxl implementation. I've not used it but in general I've used the following:

tokenizer = AutoTokenizer.from_pretrained(model_name)
config = AutoConfig.from_pretrained(model_name)
config.output_hidden_states = True
model = AutoModel.from_pretrained(model_name, config=config)
model.eval()

def cls_embedding(example):
    inputs=tokenizer(example, return_tensors='pt')
    outputs = model(**inputs)
    return outputs["hidden_states"][-1][-1][0].detach().numpy()

Note you probably want outputs["hidden_states"][-1][-1][:] if you want an embedding per input token (i.e. contextual embedding)?