How do LLMs transform tokens into vectors?

Question

I know about tokenization algorithms like BPE and some other basics of tokenization from the Hugging Face course. I've also heard about word2vec and other algorithms for assigning words to vectors. I'm very confused about how these two fit together, if at all, in LLMs. What are some common practices for converting tokenized text into input tensors?

I believe one approach is to randomly initialize a matrix of shape (embedding dimension, token vocabulary size). Multiplying with the one-hot encoded column vector for a given token will select a particular column of the matrix; thus the matrix functions as a lookup table that assigns vectors to tokens in the vocabulary. During training, update this matrix normally via backpropagation. (This approach makes no use of word2vec.)

Answer 1

Yes, it is exactly as you say. LMs (and machine translation models, too) start with a randomly initialized embedding matrix, which is learned via standard backpropagation. It is typically not implemented via multiplying one-hot vectors, which would waste memory. All deep learning frameworks have embedding layers that do an index-based lookup.