All Questions
Tagged with transformers machine-learning
54 questions
1
vote
1
answer
1k
views
Levenshtein/Edit Distance as a loss function for sequence transformer models?
Often, the loss function used for a sequence is cross entropy loss between $y_{true}$ and $y_{pred}$ where both are of size $SeqLength \times NumClasses$. When $y_{pred}=y_{true}$ we get the lowest ...
CommunityBot
- 1
1
vote
1
answer
215
views
Machine learning model for matching records
I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm ...
CommunityBot
- 1
9
votes
2
answers
4k
views
Why transformer in deep learning is called transformer?
Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
- 288
4
votes
1
answer
1k
views
Loss function for vectors when magnitude of elements and their position are both important
Context:
I am using a transformer for time series prediction.
The target and predicted tensors are both of size (8, 10, 181) which represents (batch_size, number of ...
- 1,099
6
votes
2
answers
10k
views
What is the length limit of Transformers?
From hugginface documentation:
Transformer-based models are unable to process long sequences due to their self-attention operation
How long is "long" here? 1000 inputs? 10,000 inputs?
Just ...
- 11
0
votes
0
answers
30
views
Regression on SQL query texts. Good ML model architecture
Fast regression on SQL queries. Good ML model architecture.
Our goal is to predict which SQL engine (there are 2 currently) will be faster to execute a given query.
The input is the query text and in ...
- 141
0
votes
0
answers
35
views
Why learn an embedding before self attention when training transformers?
I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
0
votes
1
answer
108
views
Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?
My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length.
However, an issue ...
- 156
0
votes
0
answers
26
views
Where does the equation $ C = 6 \times N \times T $ come from for Large Language Models, especially with a simple explanation for both passes?
Why $ C = 6 \times N \times T $?
I'm trying to understand the computational steps specifically during the backward pass of neural networks in relation to the widely cited formula ( C = 6 \times N \...
- 82.8k
13
votes
4
answers
4k
views
Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?
Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar ...
- 94k
5
votes
2
answers
725
views
Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding
In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).
This way, each cell contains semantic and ...
- 156
0
votes
0
answers
192
views
Transformer with just one input vector
I have a problem where I am mapping from 1D input sequences of length L to 1D output sequences, also of length L. These sequences contain numerical data. The input sequence is the time evolution of a ...
0
votes
1
answer
40
views
Does a model learn the same attention scores when retrained?
As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers?
It feels ...
- 990
0
votes
0
answers
328
views
block_size in transformers: does it dictate effective context length in LLMs?
I would like to understand how the block_size parameter in the huggingface transformers library works, particularly in comparison with model_max_length. I am interested in models being able to attend ...
- 453
-1
votes
2
answers
2k
views
Should I use transformer.fit_transform(X_test, y_test) or not?
tl-dr: The function model.fit() is different from transformer.fit(). My idea is to make all transformations needed on the training set and after that on the test set with fit_transform in both.
Hi!
I'...
- 103
1
vote
1
answer
450
views
Why do we mask input tokens for the decoder in a transformer? [closed]
I am currently trying to understand how the Transformer Architecture created by Vaswani et al. in 2017 works. Regarding this I have problems understanding the training process of the decoder.
If the ...
- 990
3
votes
1
answer
1k
views
Would it make sense to use group norm with transformers?
Transformers commonly use layer normalization, as explained here: Why do transformers use layer norm instead of batch norm?.
One of the arguments in that post is that batch normalization is not used ...
- 2,137
2
votes
1
answer
619
views
Time Series Classification using Transformer Encoder
Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For ...
- 1,783
2
votes
0
answers
29
views
Contrast Self attention and Attention in machine learning for non technical person?
How would you contrast Self attention and Attention in machine learning for a non technical person?
I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
- 1,021
1
vote
1
answer
1k
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
- 9,001
2
votes
1
answer
1k
views
Why does the masked language modelling (MLM) task produce useful embeddings?
Masked language modelling is the standard way of training a language model such as a transformer. Each input token has some probability (e.g. 15%) of being replaced with a <MASK> token. The ...
CommunityBot
- 1
5
votes
2
answers
10k
views
Self Attention for Variable Length Sequence Classification
I have a problem that is not particularly unique, but I'm still having trouble to figure out exactly how it's usually done.
My training set is of the form $\mathcal{T}=\{(t_i\in \mathbb{R}^{[n,m]\...
- 19.5k
3
votes
1
answer
2k
views
How does inference work on a Transformer?
(Let's say I trained a transformer for translation.)
In the training, the output sentences are given and fed into the decoder as a whole. However, with inference, only a start-of-sentence (SOS) token ...
CommunityBot
- 1
6
votes
1
answer
14k
views
Where is dropout placed in the original transformer?
I wanted to know where dropout was placed in the original transformer. According to the original paper Attention Is All You Need they say:
...
- 7
2
votes
1
answer
769
views
How do LLMs transform tokens into vectors?
I know about tokenization algorithms like BPE and some other basics of tokenization from the Hugging Face course. I've also heard about word2vec and other algorithms for assigning words to vectors. I'...
- 131k
13
votes
1
answer
5k
views
Formula to compute approximate memory requirements of Transformer models
I would like to roughly estimate the memory requirement of training an arbitrary Transformer model $M$, with $l$ layers, $h$ attention heads, an embedding dimension of $d$, and an input dimension of $...
- 103
1
vote
1
answer
2k
views
In Transformers, for the maximum length of Encoder's input sequences and Decoder's input sequences - should they be two different numbers?
I noticed that there's usually a limit for the input length of transformers. But considering there are actually two input layers - one for the Encoders and one for the Decoders, can we and should we ...
- 3,459
1
vote
1
answer
1k
views
Do BERT word embeddings change after training, depending on context?
Before answering "yes, of course", let me clarify what I mean:
After BERT has been trained, and I want to use the pretrained embeddings for some other NLP task, can I once-off extract all ...
- 256
0
votes
1
answer
402
views
bos_token for a custom Transformer
I am trying to use a Transformer to solve a time-series problem. I built the model using the Pytorch library. And I am planning to train the model from scratch. The model is looking back last L time-...
- 3,459
3
votes
1
answer
131
views
What is the best method to find the sentences that answer a given question in a document?
Let's say I have a long document (let's say 10 pages) and I have a question about the content of the document. An example could be:
Document: History of Flowers
Question: What type of flowers have ...
- 1,284
4
votes
1
answer
4k
views
Training Transformers: self attention weights vs embedding layer
I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
- 316
0
votes
1
answer
93
views
Cross lingual transfer for summarisation using XLM-R
I have a question. There's a library (uses this paper) which suggests in its cross lingual part that if the XLM-R is trained in english dataset, it can be directly applied to datasets in other ...
- 3,459
0
votes
1
answer
167
views
Is it valid to calculate a transformer neural network loss with respect to one element of a sequence input?
Suppose one sample of my training data consists of a sequence with $n$ elements. My task is to do binary classification on one element in the sequence, and my labels are such that for each sequence in ...
- 173
2
votes
1
answer
403
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 3,459
5
votes
1
answer
2k
views
BERT MLM - 80% [MASK], 10% random words and 10% same word - how does this work?
I have noticed that (from the original BERT paper) in the MLM training procedure, the authors decide to mask 15% of the words in a sentence. The mask works as following:
The masked words are ...
- 9,001
0
votes
1
answer
3k
views
Positional Encoding and Fourier transforms
While I was studying Positional Encoding, I came across an article that links coding resolution to Fourier transforms:
"For anyone who has studied finite
Fourier transforms, this problem should ...
- 668
1
vote
0
answers
150
views
How to handle problem of different random seeds giving drastically different test scores in machine learning model?
For a rigorous empirical analysis, I am training a model with three different seeds - 0, 1 and 2. In each case, I found that the model obtained through early stopping (lowest validation loss) had an ...
- 143
3
votes
2
answers
2k
views
Does dropout have any benefits when overfitting isn't a concern?
I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training ...
- 131
1
vote
0
answers
683
views
Multi-Head Attention in ViT
I need help to understand the multihead attention in ViT. Here's the code I found from GitHub:
...
2
votes
0
answers
622
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
1
vote
1
answer
2k
views
Intuition about the application of padding masks and look-ahead masks in Transformer's encoder/decoder
From the Tensorflow tutorial, the shape of the padding mask is (batch_size, 1, 1, seq_len) and look-ahead mask is ...
- 33
1
vote
3
answers
2k
views
transformer, linear layer after attention
My question regards this image:
It seems that after the multi head attention there is a linear layer as they mention also from here:
the linearity is given by the weights W^{o}. my quesion is: for ...
- 26
3
votes
1
answer
65
views
Operation modes in neural turing machine (Graves, 2014)
I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out.
More ...
- 26.5k
0
votes
0
answers
51
views
Why is self-attention used for image classification?
I'm wondering why you would use Self-Attention across an entire image for image classification. What are the advantages of Self-Attention compared to a pure MLP?
- 82.8k
2
votes
1
answer
59
views
In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
- 3,459
1
vote
1
answer
1k
views
How does Transformer use BPE?
I'm trying to re-implement Transformer on my own but I'm struggling on the input encoding.
It seems that the original paper(Attention is All You Need) mentioned that they use Byte-pair Encoding to ...
- 6,912
1
vote
0
answers
905
views
What exactly does transformer encoder + linear layer return?
I was following a Pytorch tutorial on transformers in language modelling ( https://pytorch.org/tutorials/beginner/transformer_tutorial.html ) and I came across a bunch of questions.
My goal is to make ...
- 11
1
vote
0
answers
125
views
What is the effect of changing the weight decay and warm-up steps in fine-tuning PEGASUS?
I am fine-tuning PEGASUS model using this script.
I am currently using the SAMSum dataset and I have reached a point in which the output doesn't get better.
Examples:
The Actual Summary
Alexis and ...
- 11
2
votes
1
answer
349
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 9,001
1
vote
0
answers
97
views
NLP transformers and classification interpretation
I have an NLP transformer output and build logistic regression on top of it to classify negatives and positives.
How can I see how the prediction was made on a large set of data? What if I use a ...
- 9,001