Word Embeddings with nn.Embedding except glove

Question

I'm not an expert in nlp but I know we usually use glove, word2vec or fasttext ect to get embedding vectors so what is nn.Embedding and what does it do? I mean shouldn't it be specialized for each language (dictionary of use) btw are glove and others specialized for an specific dictionary? if it is so how an general embedding (nn.Embedding) useful?

Answer 1

nn.Embedding is a generic way to allow for learning an embedding for arbitrary discrete data. This can be useful for encoding other types of categorical variables. For example you might have make/model information for cars and want a representation that allows for having the embedded representations for "Ford F-150" and "Dodge Ram" be similar since they are both pickup trucks. This is not so much about NLP, there is no grammar involved, just a way of representing this type of information.

Answer 2

Adding to Dave's answer with my NLPer perspective.

I know we usually use glove, word2vec or fasttext ect [sic] to get embedding vectors

These are different models for learning word embeddings. There's nothing language-specific about any of these yet$^*$. But it's common to pre-train word embeddings on a large corpus (in the language you care about) with one of these models. After that, you'd import the already-learned embeddings into your model. This is a form of transfer learning.

Of course, you don't have to do this. Using the process Dave describes, you could learn embeddings for the words from your task-specific data.

shouldn't it be specialized for each language (dictionary of use)

Yes, in that you wouldn't learn French GloVe embeddings then expect it to transfer well into your model for English coreference resolution. But this is, again, about using pre-trained embeddings.

are glove and others specialized for an [sic] specific dictionary?

Also yes. You choose the dictionary when training GloVe (or whichever). You need to ensure that you use the same dictionary when copying these vectors into your new model.

how an general embedding (nn.Embedding) useful? [sic]

You may not be embedding text. (See Dave's example.) You can use embeddings for any discrete data. If you are embedding text, you may not want to use a pre-trained set of word vectors. Perhaps there is domain-specific vocabulary that wasn't in the pre-trained model, or perhaps you don't trust the pre-trained model to improve your downstream performance. It's up to you as a practitioner to decide.

---

$^*$That's not to say that these are language-agnostic. Each relies on certain assumptions about the nature of the language. For instance, FastText's assumption about character n-grams would not work well for ideographic languages.