# Questions tagged [transformers]

Transformers are part of a deep learning model family using the attention mechanism to learn how to make prediction over sequential data without the need for recursion.

170 questions
Filter by
Sorted by
Tagged with
5 views

### How are the inputs fed into the decoder-only transformers during inference?

In decoder-only transformers for token prediction, I understand that during training, a whole sequence can be fed into the transformer and processes though layers. But during inference (when we want ...
8 views

### Measuring statistical difference across two distributions of attention scores

For a study I'm conducting, I have two distributions of attention scores generated via a trained transformer model. I'm plotting the two distributions and I would like to run a statistical test to ...
• 637
13 views

### Got numerical difference between two implementations

I've been working around RetNet (Paper: https://arxiv.org/pdf/2307.08621, PyTorch implementation: https://github.com/Jamie-Stirling/RetNet/). I rewrote the some of the code with TensorFlow: ...
19 views

### Transformer model conditional probability distribution of sub-sentences

I have a simple transformer model (decoder only) which is trained on some dataset containing sentences to do next-word prediction. The model captures a probability distribution $P_{\theta}(\mathbf{a})$...
9 views

### Why learn an embedding before self attention when training transformers?

I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
80 views

### What advantage do sinusoidal positional encodings have over binary positional encodings in transformer LLMs?

I've recently come across an article that discusses the reasons why large language models use sinusoidal functions to generate positional encodings — as per the famous paper Attention Is All You Need (...
20 views

### Where does the equation $C = 6 \times N \times T$ come from for Large Language Models, especially with a simple explanation for both passes?

Why $C = 6 \times N \times T$? I'm trying to understand the computational steps specifically during the backward pass of neural networks in relation to the widely cited formula ( C = 6 \times N \...
• 6,926
1 vote
10 views

### Classification of intervals in time series data of multiple instances

I have a problem that I am trying to frame. I have signal data from ECG (a classic signal over time data). A close example here: https://github.com/jjongjjong/ECG_segmentation_1DUnet I am basically ...
• 11
145 views

### Transformer with just one input vector

I have a problem where I am mapping from 1D input sequences of length L to 1D output sequences, also of length L. These sequences contain numerical data. The input sequence is the time evolution of a ...
27 views

### Does a model learn the same attention scores when retrained?

As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers? It feels ...
• 11
24 views

### Why are numbers tokenized and/or embedded before their input into transformers?

For all transformer models that are either trained for text tasks or specialized for time series or simple equation solving, they tokenize and embed numbers rather than using their raw representation. ...
• 236
1 vote
18 views

### BERT eval loss increase while performance metrics also increase

I want to fine-tune BERT for Named Entity Recognition (NER). However, when fine-tuning over several epochs on different datasets I get a weird behaviour where the training loss decreases, eval loss ...
130 views

### Is it necessary to use attention mask for mean pooling for BERT?

I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
224 views

### Why the standard deviation of the BERT weight initialization is 0.02 by default

The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers. Initializing ...
• 1,548
35 views

### Transformer model for series classification

the title more or less says it all. I want to use the transformers architecture, i.e. a transformer model, for series classification. I don’t want to predict time series, just to classify them. The ...
71 views

### Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?

My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length. However, an issue ...
209 views

### block_size in transformers: does it dictate effective context length in LLMs?

I would like to understand how the block_size parameter in the huggingface transformers library works, particularly in comparison with model_max_length. I am interested in models being able to attend ...
• 453
19 views

### How does the model learn the weights and biases in a Transformer NN architecture? [duplicate]

In the transformer block between Input Embedding Matrix and Layer normalization, where the data is scaled to mean = 0, and std = 1. How does the NN learn weights and biases and applying them to the ...
• 560
1 vote
28 views

### Why is there a residual connection around each attention layer followed by a layer normalization step in the in the decoder network? [closed]

I got the following question in a quiz, and it seems that my answer is not correct. I don't understand why: ...
• 145
1 vote
288 views

### Why do we mask input tokens for the decoder in a transformer? [closed]

I am currently trying to understand how the Transformer Architecture created by Vaswani et al. in 2017 works. Regarding this I have problems understanding the training process of the decoder. If the ...
• 13
89 views

### What embeddings are used in decoder-only models like GPT?

Decoder-only models do not use an encoder. Hence, they do not get the embedding from it. I went through this nice description of a decoder-only transformer -based model I do understand the training ...
• 55
74 views

### Transformers: Cross Attention Tensor Shapes During Inference Mode

Using the "classic" transformer model describing in "Attention is All You Need", I'm struggling to understand how the Encoder output is used by the Decoder during cross attention ...
393 views

### What are the layer normalization dimensions in transformer?

In transformer training, the activations have three dimensions: batch, feature (i.e. embedding) and time (i.e. token). Layer normalization is applied, calculating statistics (mean, standard deviation) ...
• 1,163
28 views

### Why relu not used after the second FCL in the feed forward neural network part of the transformer

I found that after the second fully connected layer of feed forward neural network, transformer doesn't apply ReLU on it. If the purpose of ReLU after the first FCL, the ReLU should be applied after ...
34 views

### Aren't layer normalizations on ViTs acting just like batch normalization? [duplicate]

I was checking some ViTs implentation like this. And, as Layernorm (LN) are assigned by channels like ln1 = nn.LayerNorm(768), and the data input of this layer has ...
34 views

### LLMs' latency and their usability for inference

I am trying to use a transformer decoder (LLM, for simplicity) to label a collection of texts, later to be used for training a classifier. I tried multiple 7B models, which I can save on my local ...
• 335
28 views

### Contrast Self attention and Attention in machine learning for non technical person?

How would you contrast Self attention and Attention in machine learning for a non technical person? I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
• 1,011
537 views

### Time Series Classification using Transformer Encoder

Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For ...
131 views

### Embeddings/tokenizers for a transformer with binary-valued data

I'm trying to train an encoder-decoder transformer model for completion of binary-valued data. The each input is basically a length-n bitstring $x = (x_1, \dots, x_n) \in \{0,1\}^n$, generated ...
• 163
347 views

### Difference between multi-head and single-head attention

Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
35 views

### Classification in BERT - why not use class as a feature?

I am currently following this post, which details how BERT was trained. I had a few questions about the classification task: In the post, it mentions that the authors of BERT decided to add ...
• 319
316 views

### Why not use input padding in the first attention block in transformer decoder

I was studying the transformer decoder code below in Keras/Tensorflow. It was not clear how they made making decisions. In the first attention block below (self.attention_1), why did they use ...
81 views

### Alternatives to classical softmax for better aggregation

I am doing machine learning on phylogenetic trees. Species within the trees have several relations, such as living at the same time or not, and I would like to use a global attention mechanism which ...
• 43
27 views

### How is metadata represented in sentiment analysis?

There are papers on semantic analysis using metadata such as "Sentiment Classification on Steam Reviews" (https://cs229.stanford.edu/proj2017/final-reports/5244171.pdf) and "Detecting ...
47 views

### How can I compute prediction explainability (e.g. SHAP) for a classifier trained on dense embeddings (SBERT)?

I have a multiclass classification problem (intent classification). I trained an XGBoost model on a dataset that was feature extracted using SBERT embeddings. I'm trying to compute an explainability ...
• 169
82 views

### How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
• 657
45 views

### Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"

I am fairly new to Transformers thus this schema confuses me a lot. I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
1 vote
163 views

### Machine learning model for matching records

I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm ...
• 33.5k
1k views

### In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?

We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as $$\alpha = Softmax(QK^T)$$ In the case of ...
1 vote
197 views

### How to determine EC2 instance type and memory for LLM inference endpoint [closed]

I am trying to estimate the costs required for hosting a fine tuned large language model for real time inference. There will be 100s of users querying the endpoint concurrently for multiple use cases ...
• 111
172 views

### Why the positional embeddings for different positions are not confined

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension))) we know <...
1 vote
97 views

### Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?

In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
• 485
179 views

### Why the positional embeddings for a specific position i, for each embedding element is different

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension))) we know <...
456 views

### Necessity of keys and queries in a decoder-only transformer model

when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$. However, in the next step, we compute a dot product on the embedding ...
• 43
107 views

### Why does ViTForMaskedImageModeling not construct the original image correctly?

I was trying to use masked image modeling in huggingface and I saw ViTForMaskedImageModeling in their documentation but I did not understand how it reconstructs the original image ...
• 1,382
1 vote
606 views

### Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?

I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part. In attention we use softmax ...
• 1,382
259 views

### Effect of dK (key vector dimension) in transformers

Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers? My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
1 vote
341 views

### Why the Transformer model does not require negative sampling but word2vec does?

Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side. For word2vec models, negative sampling is used for computational reasons: Is negative sampling ...
• 2,435
194 views

### Expectation of the absolute value of the product of correlated jointly gaussians?

I am reading the Performer paper https://arxiv.org/abs/2009.14794. To understand their ReLU kernel used to approximate softmax attention, I need to evaluate $\mathbb{E}[ReLU(x^T w) \cdot ReLU(y^T w)]$ ...
• 23