2 votes

Why not perform weight decay on layernorm/embedding?

After some searching, I find some explanations: karpathy's explanation and a disscusion in the pytorch forum. @morganmcg1 the purpose of L2 regularization is to "spread out" the weights in ...
user avatar
1 vote
Accepted

Text similarity for badly written text

Yes, there’s a lot of research done on recovering from misspellings. The seminal probabilistic framework is Probability scoring for spelling correction by Church and Gale, which takes a Bayesian noisy ...
user avatar
1 vote

Which metric to use for language translation?

Answer As per my comment above: You can only know if the output is correct by having a reference sentence for the output. Comparing the input and output for the accuracy of the output won't work. ...
user avatar
  • 3,466
1 vote
Accepted

In a tranformer, the same word can have different attention weights in different sentences?

Yes, and it is not only the case for Transformer but for nearly any deep learning NLP model. Only when treating natural language data as bag-of-words, the sentence is considered as a sum of ...
user avatar
  • 113k
1 vote
Accepted

BERT MLM - 80% [MASK], 10% random words and 10% same word - how does this work?

The answer to your question is in §3.1 of the paper. First, bear in mind that only the “masked” tokens (about 15%) are predicted during training, not all tokens. With that in mind, I would teach it ...
user avatar
1 vote

What is the role of feed forward layer in Transformer Neural Network architecture?

Consider encoder part of transformer. If there is no feed-forward layer, self-attention is simply performing re-averaging of value vectors. In order to add more model function, i.e. element-wise non-...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible