All Questions
Tagged with language-models attention
2 questions
1
vote
1
answer
94
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
2
votes
1
answer
830
views
BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?
I'm reading the BERT paper and jalammar's illustrative guide for BERT.
I don't understand 2 things about the method's crux - the masked language model:
why does masking requires us to sample (take ...
- 3,360