All Questions
Tagged with transformers language-models
4 questions with no upvoted or accepted answers
2
votes
0
answers
623
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
1
vote
0
answers
91
views
What's the role of masking in transformer and BERT?
I've recently implemented the architectures of Transformer and BERT and found that they both have common property - masked layers among one of them. I have come to questions like below.
As far as I ...
- 123
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
0
votes
0
answers
47
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 319