All Questions
20 questions
0
votes
0
answers
30
views
Regression on SQL query texts. Good ML model architecture
Fast regression on SQL queries. Good ML model architecture.
Our goal is to predict which SQL engine (there are 2 currently) will be faster to execute a given query.
The input is the query text and in ...
- 141
0
votes
0
answers
35
views
Why learn an embedding before self attention when training transformers?
I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
0
votes
0
answers
26
views
Where does the equation $ C = 6 \times N \times T $ come from for Large Language Models, especially with a simple explanation for both passes?
Why $ C = 6 \times N \times T $?
I'm trying to understand the computational steps specifically during the backward pass of neural networks in relation to the widely cited formula ( C = 6 \times N \...
- 7,074
1
vote
1
answer
1k
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
2
votes
1
answer
770
views
How do LLMs transform tokens into vectors?
I know about tokenization algorithms like BPE and some other basics of tokenization from the Hugging Face course. I've also heard about word2vec and other algorithms for assigning words to vectors. I'...
- 133
1
vote
1
answer
2k
views
In Transformers, for the maximum length of Encoder's input sequences and Decoder's input sequences - should they be two different numbers?
I noticed that there's usually a limit for the input length of transformers. But considering there are actually two input layers - one for the Encoders and one for the Decoders, can we and should we ...
- 2,485
0
votes
1
answer
402
views
bos_token for a custom Transformer
I am trying to use a Transformer to solve a time-series problem. I built the model using the Pytorch library. And I am planning to train the model from scratch. The model is looking back last L time-...
- 113
1
vote
1
answer
1k
views
Do BERT word embeddings change after training, depending on context?
Before answering "yes, of course", let me clarify what I mean:
After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
3
votes
1
answer
131
views
What is the best method to find the sentences that answer a given question in a document?
Let's say I have a long document (let's say 10 pages) and I have a question about the content of the document. An example could be:
Document: History of Flowers
Question: What type of flowers have ...
4
votes
1
answer
4k
views
Training Transformers: self attention weights vs embedding layer
I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
- 143
0
votes
1
answer
94
views
Cross lingual transfer for summarisation using XLM-R
I have a question. There's a library (uses this paper) which suggests in its cross lingual part that if the XLM-R is trained in english dataset, it can be directly applied to datasets in other ...
- 101
2
votes
1
answer
1k
views
Why does the masked language modelling (MLM) task produce useful embeddings?
Masked language modelling is the standard way of training a language model such as a transformer. Each input token has some probability (e.g. 15%) of being replaced with a <MASK> token. The ...
- 1,203
5
votes
2
answers
730
views
Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding
In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).
This way, each cell contains semantic and ...
- 361
2
votes
1
answer
404
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 465
5
votes
1
answer
2k
views
BERT MLM - 80% [MASK], 10% random words and 10% same word - how does this work?
I have noticed that (from the original BERT paper) in the MLM training procedure, the authors decide to mask 15% of the words in a sentence. The mask works as following:
The masked words are ...
- 153
2
votes
0
answers
623
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
13
votes
4
answers
4k
views
Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?
Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar ...
- 7,074
1
vote
0
answers
905
views
What exactly does transformer encoder + linear layer return?
I was following a Pytorch tutorial on transformers in language modelling ( https://pytorch.org/tutorials/beginner/transformer_tutorial.html ) and I came across a bunch of questions.
My goal is to make ...
- 11
2
votes
1
answer
349
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 23
0
votes
0
answers
26
views
how to classify a part of sentence using transformer
The problem is like this: I have many sentences and for each sentence I have a continuous sequence of words that I want to mask out. Then I want to train a model to classify the masked part to 1/0.
...
- 133