All Questions
Tagged with language-models transformers
8 questions
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
0
votes
0
answers
47
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 319
0
votes
1
answer
2k
views
Difference between "transfer learning" and "fine-tuning"
I am currently reading the BERT paper and it splits the use of pre-trained models into two categories:
Feature-based whereby you just take the embeddings for the tokens and plug it into whatever ...
- 1,046
2
votes
1
answer
403
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 465
1
vote
0
answers
91
views
What's the role of masking in transformer and BERT?
I've recently implemented the architectures of Transformer and BERT and found that they both have common property - masked layers among one of them. I have come to questions like below.
As far as I ...
- 123
2
votes
0
answers
622
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
2
votes
1
answer
349
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 23
8
votes
1
answer
573
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,347