What embeddings are used in decoder-only models like GPT?

Question

Decoder-only models do not use an encoder. Hence, they do not get the embedding from it.

I went through this nice description of a decoder-only transformer -based model

I do understand the training phase but need some clarification regarding one thing in particular.

1. What is the word embedding that is being used?
2. Does it start with a usual word embedding like Glove or word2vec and then update it through training?
3. If so how is that done? During inference what is the word embedding that's used?

It would be nice if you gave an example as well, like GPT.

Answer 1

1. Embeddings are trained from scratch. I think of it as a hash map / dict in python that maps token to a 1d vector (embedding of a token). They are initialized from noise.
2. During the training phase the backpropagation algorithm computes loss with respect to those embeddings and that is how they being trained.
3. During inference the embeddings are frozen. Imaging you are given a sentence, it first is split into tokens by tokenizer, then those tokens are converted into embeddings using the "dict" I mentioned earlier.

The fact that in classic algorithms such as word2vec there are two sets of embeddings might confuse you as there is only one set in transformers.