How to compute the start and end vector for BERT question answering?

Question

Fine tunning BERT for the question answering task requires the training of a start vector and an end vector as it is said in the original paper : https://aclanthology.org/N19-1423/, see below

I perfectly understand, once you have those two vectors, how you extract the answer within the context. However, authors do not say much about how these vectors are trained. It looks like a classic embedding like tokens embeddings since it seems that there is a unique start token and a unique end token. But if so, how come this work? How one representation of the start vector could ever work for all pairs of (question,context)? Why those vectors do not depend on the very specific input pairs?

Answer 1

The start vector ($S$) contains a set of weights. You can implement it as a linear layer without bias. Remember dot product is just element wise multiplication and sum. This is what a linear layer without a bias does. In other words you use an nn.Linear() layer in Pytorch where input_features equals the size of $T_i$ and output_features is 1. And your start vector $S$ is the weights of this linear layer. The linear is applied at all the token locations (or in other words, a dot product between $S$ and all $T_i$s are taken, yielding a scalar value at each token location $i$) You apply Softmax on top of it. The softmax scores can be interpreted as the probability of a particular token being start token. At the time of training, you compute cross entropy between the softmax scores and a one hot vector that has a 1 at the location of the correct start token.

For the end token, loss is computed in similar fashion using a learnt end token vector. And the total loss is the sum of the cross entropy loss for start token and end token.

At the time of finetuning you find the most probable start and end token with a constraint that the position/order of the end token must be greater than or equal to the position of the start token

References

1. https://mccormickml.com/2020/03/10/question-answering-with-a-fine-tuned-BERT/
2. https://d2l.ai/chapter_natural-language-processing-applications/finetuning-bert.html

Answer 2

I was wondering the same question and my guess (inspired by the answer above) is that the start/end vector is unique (after model training) and it is probably a linear layer.

How it works (still a guess):

• The input is a concatenation between question and context.
• Inside the BERT structure, each token in the context can attend to the tokens of the question, i.e. the output embedding contains knowledge of the question.
• The unique start/end vector simply extract the information from the output embedding by dot-product.