2
$\begingroup$

The thought just occurred to me while I was processing data. If we're using the [CLS] token for classification, then it would obviously make sense to include it, but if we're not using that token do we have to include it?

For example, if I'm performing a task where I just need the positions of specific tokens excluding the [CLS] and [SEP] tokens, then it seems to me that including the two tokens would be unnecessary. I've heard the opinion that "it'd be convenient to include them" since it would ensure that the pretraining and fine-tuning settings would be the same, but I'm not exactly convinced by that argument and was wondering if anyone more knowledgeable may have something to comment.

Thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

BERT was trained with the special tokens, so it expects them to be on the input. Analyses of BERT's self-attention (e.g., Clark et al., 2019; Voita et al., 2019) show that the positions corresponding to special tokens are often used by the self-attention, probably having some technical function.

It is certainly possible to finetune BERT not to do so (or trained BERT from scratch without them), but it would only make the fine-tuning more difficult with more new things to learn for the model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is there any proof or research regarding this? When I run models without the special tokens the performance seems fine (although there is the possibility that performance may be even better if I were to include them) and I'm wondering if there's any detailed analysis that I could read up on. $\endgroup$
    – Sean
    Commented Apr 15, 2021 at 8:14
  • $\begingroup$ Jindřich makes a good point: it probably just makes the learning harder. I doubt such a detailed analysis exists, because it seems extremely marginal amongst the problems in NLP, and because of the depth+uninterpretability of the model, it’s hard to give a more meaningful resolution than “fine-tuning on this data, we see this score versus that score”. If it’s sufficiently important to you, you can be the pioneer that studies it. $\endgroup$
    – Arya McCarthy
    Commented Apr 15, 2021 at 8:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.