0
$\begingroup$

I have tried to search from a few sources, but I did not see any one of them specifically talking about this issue. For example This blog post seems to imply that the embedding used in transformer is learned separately (in other words fixed during transformer training.

As is the case in NLP applications in general, we begin by turning each input word into a vector using an embedding algorithm.

but it is rather vague. But in nano GPT code it seems embedding layer is part of the model, which means it will be updated during the model training. So which case is it? Is the embedding for GPT a pre-processing stage or part of the trainable parameters?

$\endgroup$
3
  • 2
    $\begingroup$ I don't see where in the blog post that what you describe is implied; it would help to edit the question to include a quote so that we can clarify it. $\endgroup$ Sep 7, 2023 at 16:55
  • $\begingroup$ @AryaMcCarthy done. and there is a link from that statement which tells you what 'algorithm' it is referring to. The fact that it is a separate algorithm heavily implies it is a pre-processing procedure. $\endgroup$
    – Sam
    Sep 9, 2023 at 0:51
  • 2
    $\begingroup$ Thanks for selecting the quote. The quote doesn't actually entail that it's pre-processing. (Admittedly, the embedding "algorithm" isn't terribly sophisticated. A lookup table of word to index, then slicing out a row in a matrix.) So I wouldn't stress about it. GPT embeddings are learnable parameters. $\endgroup$ Sep 9, 2023 at 3:28

1 Answer 1

3
$\begingroup$

Speaking for GPT1-3, the models learn an embedding. These models follow the paper Attention Is All You Need which states that they "use learned embeddings to convert the input tokens and output tokens to vectors of dimension d_model".

Skim the papers for GPT 1,2 and 3 to see that GPT3 follows GPT2, GPT2 follows GPT and GPT uses the architecture from Attention Is All You Need. Looking at the code for gpt2 may also help you understand what is going on under the hood.

$\endgroup$
1
  • $\begingroup$ yeap. I referred to the nanoGPT paper and saw the embedding layer is trainable. But if you look at the specific blog post I mentioned it seems to suggest otherwise. I am trying to reconcile this two conflicting sources of information. $\endgroup$
    – Sam
    Sep 9, 2023 at 0:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.