Timeline for Why BERT keep some masked tokens unchanged?
Current License: CC BY-SA 4.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 15, 2021 at 14:47 | history | edited | Lerner Zhang | CC BY-SA 4.0 |
added 8 characters in body
|
| Apr 18, 2020 at 13:34 | vote | accept | eric2323223 | ||
| Apr 18, 2020 at 13:34 | history | bounty ended | eric2323223 | ||
| Apr 18, 2020 at 13:34 | comment | added | eric2323223 | Thank you for the explanation. I kind of understand that unchanged token is to let model also reference the token itself in generating its embedding. | |
| Apr 17, 2020 at 10:31 | comment | added | Lerner Zhang | I've updated my answer and elaborated on that. Just imagine that we add a random token or just a [MASK] token at the end in the input and the model will just ignore that because it just provides no information for the task. | |
| Apr 17, 2020 at 8:25 | comment | added | eric2323223 | "Without the unchanged tokens, the model would mostly just ignores the token in the mirror position when predicting that token.", could you please elaborate more,like use a concrete example? | |
| Apr 16, 2020 at 16:43 | history | edited | Lerner Zhang | CC BY-SA 4.0 |
added 514 characters in body
|
| Apr 16, 2020 at 16:36 | history | edited | Lerner Zhang | CC BY-SA 4.0 |
added 514 characters in body
|
| Apr 16, 2020 at 16:27 | comment | added | Lerner Zhang | Without the unchanged tokens, the model would mostly just ignores the token in the mirror position when predicting that token. | |
| Apr 16, 2020 at 16:21 | comment | added | Lerner Zhang | "replacing the word with random word should include replace it with the original word", you are right, but the probability would be much less than 1.5%(10% of 15%). | |
| Apr 16, 2020 at 16:10 | comment | added | eric2323223 | I've read "The purpose of this is to bias the representation towards the actual observed word" but don't quite understand. In my mind, replacing the word with random word should include replace it with the original word, because the model is making the same prediction. Could you please share your thought? @Lerner | |
| Apr 16, 2020 at 14:33 | history | answered | Lerner Zhang | CC BY-SA 4.0 |