2
votes
Why not perform weight decay on layernorm/embedding?
After some searching, I find some explanations:
karpathy's explanation and a disscusion in the pytorch forum.
@morganmcg1 the purpose of L2 regularization is to "spread out" the weights in ...
- 81
1
vote
Accepted
Text similarity for badly written text
Yes, there’s a lot of research done on recovering from misspellings.
The seminal probabilistic framework is Probability scoring for spelling correction by Church and Gale, which takes a Bayesian noisy ...
- 6,797
1
vote
Which metric to use for language translation?
Answer
As per my comment above:
You can only know if the output is correct by having a reference sentence for the output. Comparing the input and output for the accuracy of the output won't work.
...
- 3,466
1
vote
Accepted
In a tranformer, the same word can have different attention weights in different sentences?
Yes, and it is not only the case for Transformer but for nearly any deep learning NLP model. Only when treating natural language data as bag-of-words, the sentence is considered as a sum of ...
Tim♦
- 113k
1
vote
Accepted
BERT MLM - 80% [MASK], 10% random words and 10% same word - how does this work?
The answer to your question is in §3.1 of the paper.
First, bear in mind that only the “masked” tokens (about 15%) are predicted during training, not all tokens. With that in mind, I would teach it ...
- 6,797
1
vote
What is the role of feed forward layer in Transformer Neural Network architecture?
Consider encoder part of transformer.
If there is no feed-forward layer, self-attention is simply performing re-averaging of value vectors.
In order to add more model function, i.e. element-wise non-...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible