Why BERT use learned positional embedding? Compared with sinusoidal positional encoding used in Transformer, BERT's learned-lookup-table solution has 2 drawbacks in my mind:


*

*Fixed length

*Cannot reflect relative distance


Could anyone please tell me the considerations behind such design?
 A: Here is my current understanding to my own question.
It probably related BERT's transfer learning background. The learned-lookup-table indeed increase learning effort in pretrain stage, but the extra effort can be almost ingnored compared to number of the trainable parameters in transformer encoder, it also should be accepted given the pretrain stage one-time effort and meant to be time comsuming.
While in the finetune and prediction stages, it's much faster because the sinusoidal positional encoding need to be computed at every position.
A: 

*

*Fixed length


BERT, same as Transformer, use attention as a key feature. The attention as used in those models, has a fixed span as well.



*Cannot reflect relative distance


We assume neural networks to be universal function approximators. If that is the case, why wouldn't it be able to learn building the Fourier terms by itself?
Why did they use it? Because it was more flexible then the approach used in Transformer. It is learned, so possibly it can figure out by itself something better--that's the general assumption behind deep learning as a whole. It also simply proved to work better.
