bos_token for a custom Transformer

Question

I am trying to use a Transformer to solve a time-series problem. I built the model using the Pytorch library. And I am planning to train the model from scratch. The model is looking back last L time-steps values of N data-series and should predict the next time-step (N values).

For this, I figured out the architecture, however, I am a bit puzzeled by what should I add as the output of the decoder. I found, we normaly use a bos_token (beginning of sentence token). So, is it okay if I just use a Vector of all zeros as the bos_token? Or is there any specific vector?

Answer 1

The standard way is to include the BOS token in the vocabulary, which will get an embedding like any other token in the vocabulary. During tokenization, you need to make sure that you add the token. Most of the standard tokenizers do so (e.g., Huggingface tokenizer have an add_special_tokens attribute for that).

Freezing the embedding to all zeros should also work. In such a case, the remaining embeddings will be forced to rescale, such that the BOS token has a zero embedding.