Questions tagged [transformers]
Transformers are part of a deep learning model family using the attention mechanism to learn how to make prediction over sequential data without the need for recursion.
172 questions
0
votes
0
answers
6
views
How should I input and output feature and target timeseries to timeseries transformer
I am trying out PatchTST timeseries transformer (paper, code) on a timeseries data that I have. The way PatchTST handles data is as follows:
Note that on line 78-79, the repo does following:
...
- 235
0
votes
0
answers
14
views
How to reduce prediction error in a long-term time series prediction
In this project, I aim to predict a missing segment of 100,000 points in the middle of a sequence using a total of 100,000 points before and after the gap. I have already tried polynomial fitting and ...
- 1
1
vote
0
answers
23
views
Is there a way to 'infuse' model so transformer pruning didn't have to start from scratch?
Transformers are infamous for it's massive resources need to perform an impressive feat in NLP and currently also in image processing especially image segmentation. There is a trick called soft-token ...
- 11
0
votes
0
answers
22
views
the attention map fades - is this normal?
I’m currently working on building a Vision Transformer (ViT), and so far, things seem to be going well — low loss values, high accuracy. However, when I visualized the attention maps, I noticed that ...
0
votes
0
answers
17
views
Transformers Operations Order [duplicate]
What is better ?
A) add attention, then add FFN. So I think that today's version of Transformers force the FFN to correct the Attention impacte and to do the FFN work. Maybe it's intended, but I ...
- 23
0
votes
0
answers
30
views
Regression on SQL query texts. Good ML model architecture
Fast regression on SQL queries. Good ML model architecture.
Our goal is to predict which SQL engine (there are 2 currently) will be faster to execute a given query.
The input is the query text and in ...
- 141
1
vote
1
answer
40
views
Attention, FFN Add&Norm Order
The actual structure Add to X Attention, then FFN.
I my structure on the right image, only the tiny $ \delta e$ generated by the FFN is added to ...
- 23
0
votes
0
answers
21
views
How are the inputs fed into the decoder-only transformers during inference?
In decoder-only transformers for token prediction, I understand that during training, a whole sequence can be fed into the transformer and processes though layers. But during inference (when we want ...
- 283
0
votes
0
answers
8
views
Measuring statistical difference across two distributions of attention scores
For a study I'm conducting, I have two distributions of attention scores generated via a trained transformer model. I'm plotting the two distributions and I would like to run a statistical test to ...
- 659
0
votes
0
answers
16
views
Got numerical difference between two implementations
I've been working around RetNet (Paper: https://arxiv.org/pdf/2307.08621, PyTorch implementation: https://github.com/Jamie-Stirling/RetNet/). I rewrote the some of the code with TensorFlow:
...
0
votes
0
answers
58
views
Transformer model conditional probability distribution of sub-sentences
I have a simple transformer model (decoder only) which is trained on some dataset containing sentences to do next-word prediction. The model captures a probability distribution $P_{\theta}(\mathbf{a})$...
0
votes
0
answers
35
views
Why learn an embedding before self attention when training transformers?
I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
6
votes
1
answer
173
views
What advantage do sinusoidal positional encodings have over binary positional encodings in transformer LLMs?
I've recently come across an article that discusses the reasons why large language models use sinusoidal functions to generate positional encodings — as per the famous paper Attention Is All You Need (...
- 156
0
votes
0
answers
26
views
Where does the equation $ C = 6 \times N \times T $ come from for Large Language Models, especially with a simple explanation for both passes?
Why $ C = 6 \times N \times T $?
I'm trying to understand the computational steps specifically during the backward pass of neural networks in relation to the widely cited formula ( C = 6 \times N \...
- 7,074
1
vote
0
answers
11
views
Classification of intervals in time series data of multiple instances
I have a problem that I am trying to frame. I have signal data from ECG (a classic signal over time data). A close example here: https://github.com/jjongjjong/ECG_segmentation_1DUnet
I am basically ...
- 125
0
votes
0
answers
192
views
Transformer with just one input vector
I have a problem where I am mapping from 1D input sequences of length L to 1D output sequences, also of length L. These sequences contain numerical data. The input sequence is the time evolution of a ...
0
votes
1
answer
40
views
Does a model learn the same attention scores when retrained?
As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers?
It feels ...
- 11
0
votes
0
answers
27
views
Why are numbers tokenized and/or embedded before their input into transformers?
For all transformer models that are either trained for text tasks or specialized for time series or simple equation solving, they tokenize and embed numbers rather than using their raw representation. ...
- 236
1
vote
0
answers
34
views
BERT eval loss increase while performance metrics also increase
I want to fine-tune BERT for Named Entity Recognition (NER). However, when fine-tuning over several epochs on different datasets I get a weird behaviour where the training loss decreases, eval loss ...
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
0
votes
1
answer
423
views
Why the standard deviation of the BERT weight initialization is 0.02 by default
The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers.
Initializing ...
- 1,568
0
votes
1
answer
52
views
Transformer model for series classification
the title more or less says it all. I want to use the transformers architecture, i.e. a transformer model, for series classification. I don’t want to predict time series, just to classify them.
The ...
- 101
0
votes
1
answer
108
views
Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?
My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length.
However, an issue ...
- 1
0
votes
0
answers
327
views
block_size in transformers: does it dictate effective context length in LLMs?
I would like to understand how the block_size parameter in the huggingface transformers library works, particularly in comparison with model_max_length. I am interested in models being able to attend ...
- 453
0
votes
0
answers
19
views
How does the model learn the weights and biases in a Transformer NN architecture? [duplicate]
In the transformer block between Input Embedding Matrix and Layer normalization, where the data is scaled to mean = 0, and std = 1. How does the NN learn weights and biases and applying them to the ...
- 570
1
vote
0
answers
65
views
Why is there a residual connection around each attention layer followed by a layer normalization step in the in the decoder network? [closed]
I got the following question in a quiz, and it seems that my answer is not correct. I don't understand why:
...
- 145
1
vote
1
answer
450
views
Why do we mask input tokens for the decoder in a transformer? [closed]
I am currently trying to understand how the Transformer Architecture created by Vaswani et al. in 2017 works. Regarding this I have problems understanding the training process of the decoder.
If the ...
- 13
0
votes
1
answer
139
views
What embeddings are used in decoder-only models like GPT?
Decoder-only models do not use an encoder. Hence, they do not get the embedding from it.
I went through this nice description of a decoder-only transformer -based model
I do understand the training ...
- 55
1
vote
1
answer
170
views
Transformers: Cross Attention Tensor Shapes During Inference Mode
Using the "classic" transformer model describing in "Attention is All You Need", I'm struggling to understand how the Encoder output is used by the Decoder during cross attention ...
- 171
2
votes
1
answer
585
views
What are the layer normalization dimensions in transformer?
In transformer training, the activations have three dimensions: batch, feature (i.e. embedding) and time (i.e. token). Layer normalization is applied, calculating statistics (mean, standard deviation) ...
- 1,203
-1
votes
1
answer
67
views
Aren't layer normalizations on ViTs acting just like batch normalization? [duplicate]
I was checking some ViTs implentation like this.
And, as Layernorm (LN) are assigned by channels like ln1 = nn.LayerNorm(768), and the data input of this layer has ...
- 156
2
votes
0
answers
29
views
Contrast Self attention and Attention in machine learning for non technical person?
How would you contrast Self attention and Attention in machine learning for a non technical person?
I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
- 1,021
2
votes
1
answer
619
views
Time Series Classification using Transformer Encoder
Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For ...
0
votes
1
answer
184
views
Embeddings/tokenizers for a transformer with binary-valued data
I'm trying to train an encoder-decoder transformer model for completion of binary-valued data. The each input is basically a length-n bitstring $x = (x_1, \dots, x_n) \in \{0,1\}^n$, generated ...
- 163
3
votes
1
answer
457
views
Difference between multi-head and single-head attention
Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
- 51
0
votes
0
answers
47
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 319
0
votes
1
answer
410
views
Why not use input padding in the first attention block in transformer decoder
I was studying the transformer decoder code below in Keras/Tensorflow. It was not clear how they made making decisions.
In the first attention block below (self.attention_1), why did they use ...
- 1
0
votes
0
answers
32
views
How is metadata represented in sentiment analysis?
There are papers on semantic analysis using metadata such as "Sentiment Classification on Steam Reviews" (https://cs229.stanford.edu/proj2017/final-reports/5244171.pdf) and "Detecting ...
- 1
0
votes
0
answers
64
views
Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"
I am fairly new to Transformers thus this schema confuses me a lot.
I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
1
vote
1
answer
215
views
Machine learning model for matching records
I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm ...
- 35.2k
2
votes
1
answer
1k
views
In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?
We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as
$$\alpha = Softmax(QK^T)$$
In the case of ...
- 31
1
vote
0
answers
220
views
How to determine EC2 instance type and memory for LLM inference endpoint [closed]
I am trying to estimate the costs required for hosting a fine tuned large language model for real time inference. There will be 100s of users querying the endpoint concurrently for multiple use cases ...
- 111
0
votes
1
answer
226
views
Why the positional embeddings for different positions are not confined
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
1
vote
2
answers
137
views
Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?
In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
- 485
0
votes
1
answer
208
views
Why the positional embeddings for a specific position i, for each embedding element is different
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
4
votes
2
answers
563
views
Necessity of keys and queries in a decoder-only transformer model
when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$.
However, in the next step, we compute a dot product on the embedding ...
- 43
2
votes
1
answer
129
views
Why does ViTForMaskedImageModeling not construct the original image correctly?
I was trying to use masked image modeling in huggingface and I saw ViTForMaskedImageModeling in their documentation but I did not understand how it reconstructs the original image ...
- 1,382
1
vote
1
answer
726
views
Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?
I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part.
In attention we use softmax ...
- 1,382
2
votes
1
answer
413
views
Effect of dK (key vector dimension) in transformers
Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers?
My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
1
vote
1
answer
388
views
Why the Transformer model does not require negative sampling but word2vec does?
Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side.
For word2vec models, negative sampling is used for computational reasons:
Is negative sampling ...
- 2,485