Timeline for Why BERT use learned positional embedding?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 12, 2021 at 22:28 | history | edited | Tim | CC BY-SA 4.0 |
deleted 157 characters in body
|
| Nov 12, 2021 at 22:27 | comment | added | Tim | @ondra.cifka attention by itself not, but it depends on position embeddings that do. But agree, the wording might have been confusing so I removed that part. | |
| Nov 12, 2021 at 21:36 | comment | added | ondra.cifka | The weights (parameters) of the network (i.e. the linear layers) are learned, but these do not depend on position. The attention weights (coefficients) are computed based on these parameters, so strictly speaking they are not learned, only the way to compute them is. And I don't see how this would result in it having a "fixed span". | |
| Nov 11, 2021 at 20:45 | comment | added | Tim | @ondra.cifka it has learned parameters, hence it’s “learned”. | |
| Nov 11, 2021 at 20:42 | comment | added | ondra.cifka | Your first claim is not correct. The attention weights are not learned, they are computed based on keys and queries which are different for every input, so attention can in principle generalize to different input lengths. | |
| Nov 1, 2021 at 8:29 | history | answered | Tim | CC BY-SA 4.0 |