Skip to main content

Questions tagged [transformers]

Transformers are part of a deep learning model family using the attention mechanism to learn how to make prediction over sequential data without the need for recursion.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

How are the inputs fed into the decoder-only transformers during inference?

In decoder-only transformers for token prediction, I understand that during training, a whole sequence can be fed into the transformer and processes though layers. But during inference (when we want ...
Daniel Mendoza's user avatar
0 votes
0 answers
8 views

Measuring statistical difference across two distributions of attention scores

For a study I'm conducting, I have two distributions of attention scores generated via a trained transformer model. I'm plotting the two distributions and I would like to run a statistical test to ...
James Arten's user avatar
0 votes
0 answers
13 views

Got numerical difference between two implementations

I've been working around RetNet (Paper: https://arxiv.org/pdf/2307.08621, PyTorch implementation: https://github.com/Jamie-Stirling/RetNet/). I rewrote the some of the code with TensorFlow: ...
UndefinedCpp's user avatar
0 votes
0 answers
19 views

Transformer model conditional probability distribution of sub-sentences

I have a simple transformer model (decoder only) which is trained on some dataset containing sentences to do next-word prediction. The model captures a probability distribution $P_{\theta}(\mathbf{a})$...
JazzJammer's user avatar
0 votes
0 answers
9 views

Why learn an embedding before self attention when training transformers?

I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
Nicolas Johnson's user avatar
4 votes
0 answers
80 views

What advantage do sinusoidal positional encodings have over binary positional encodings in transformer LLMs?

I've recently come across an article that discusses the reasons why large language models use sinusoidal functions to generate positional encodings — as per the famous paper Attention Is All You Need (...
Philip Voinea's user avatar
0 votes
0 answers
20 views

Where does the equation $ C = 6 \times N \times T $ come from for Large Language Models, especially with a simple explanation for both passes?

Why $ C = 6 \times N \times T $? I'm trying to understand the computational steps specifically during the backward pass of neural networks in relation to the widely cited formula ( C = 6 \times N \...
Charlie Parker's user avatar
1 vote
0 answers
10 views

Classification of intervals in time series data of multiple instances

I have a problem that I am trying to frame. I have signal data from ECG (a classic signal over time data). A close example here: https://github.com/jjongjjong/ECG_segmentation_1DUnet I am basically ...
PPenton's user avatar
  • 11
0 votes
0 answers
145 views

Transformer with just one input vector

I have a problem where I am mapping from 1D input sequences of length L to 1D output sequences, also of length L. These sequences contain numerical data. The input sequence is the time evolution of a ...
Blahblahblacksheep's user avatar
0 votes
1 answer
27 views

Does a model learn the same attention scores when retrained?

As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers? It feels ...
rick's user avatar
  • 11
0 votes
0 answers
24 views

Why are numbers tokenized and/or embedded before their input into transformers?

For all transformer models that are either trained for text tasks or specialized for time series or simple equation solving, they tokenize and embed numbers rather than using their raw representation. ...
Fine-Tuning's user avatar
1 vote
0 answers
18 views

BERT eval loss increase while performance metrics also increase

I want to fine-tune BERT for Named Entity Recognition (NER). However, when fine-tuning over several epochs on different datasets I get a weird behaviour where the training loss decreases, eval loss ...
CodingSquirrel's user avatar
0 votes
0 answers
130 views

Is it necessary to use attention mask for mean pooling for BERT?

I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
mutli-arm-bandit's user avatar
0 votes
1 answer
224 views

Why the standard deviation of the BERT weight initialization is 0.02 by default

The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers. Initializing ...
mon's user avatar
  • 1,548
0 votes
1 answer
35 views

Transformer model for series classification

the title more or less says it all. I want to use the transformers architecture, i.e. a transformer model, for series classification. I don’t want to predict time series, just to classify them. The ...
exch_cmmnt_memb's user avatar
0 votes
1 answer
71 views

Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?

My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length. However, an issue ...
ENing's user avatar
  • 1
0 votes
0 answers
209 views

block_size in transformers: does it dictate effective context length in LLMs?

I would like to understand how the block_size parameter in the huggingface transformers library works, particularly in comparison with model_max_length. I am interested in models being able to attend ...
Nucular's user avatar
  • 453
0 votes
0 answers
19 views

How does the model learn the weights and biases in a Transformer NN architecture? [duplicate]

In the transformer block between Input Embedding Matrix and Layer normalization, where the data is scaled to mean = 0, and std = 1. How does the NN learn weights and biases and applying them to the ...
kms's user avatar
  • 560
1 vote
0 answers
28 views

Why is there a residual connection around each attention layer followed by a layer normalization step in the in the decoder network? [closed]

I got the following question in a quiz, and it seems that my answer is not correct. I don't understand why: ...
Minions's user avatar
  • 145
1 vote
1 answer
288 views

Why do we mask input tokens for the decoder in a transformer? [closed]

I am currently trying to understand how the Transformer Architecture created by Vaswani et al. in 2017 works. Regarding this I have problems understanding the training process of the decoder. If the ...
dukegin's user avatar
  • 13
0 votes
1 answer
89 views

What embeddings are used in decoder-only models like GPT?

Decoder-only models do not use an encoder. Hence, they do not get the embedding from it. I went through this nice description of a decoder-only transformer -based model I do understand the training ...
tintin98's user avatar
0 votes
1 answer
74 views

Transformers: Cross Attention Tensor Shapes During Inference Mode

Using the "classic" transformer model describing in "Attention is All You Need", I'm struggling to understand how the Encoder output is used by the Decoder during cross attention ...
NickBraunagel's user avatar
2 votes
1 answer
393 views

What are the layer normalization dimensions in transformer?

In transformer training, the activations have three dimensions: batch, feature (i.e. embedding) and time (i.e. token). Layer normalization is applied, calculating statistics (mean, standard deviation) ...
Denziloe's user avatar
  • 1,163
0 votes
0 answers
28 views

Why relu not used after the second FCL in the feed forward neural network part of the transformer

I found that after the second fully connected layer of feed forward neural network, transformer doesn't apply ReLU on it. If the purpose of ReLU after the first FCL, the ReLU should be applied after ...
jungbeom Ko's user avatar
-1 votes
1 answer
34 views

Aren't layer normalizations on ViTs acting just like batch normalization? [duplicate]

I was checking some ViTs implentation like this. And, as Layernorm (LN) are assigned by channels like ln1 = nn.LayerNorm(768), and the data input of this layer has ...
Rafael Toledo's user avatar
0 votes
0 answers
34 views

LLMs' latency and their usability for inference

I am trying to use a transformer decoder (LLM, for simplicity) to label a collection of texts, later to be used for training a classifier. I tried multiple 7B models, which I can save on my local ...
David Harar's user avatar
2 votes
0 answers
28 views

Contrast Self attention and Attention in machine learning for non technical person?

How would you contrast Self attention and Attention in machine learning for a non technical person? I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
Katsu's user avatar
  • 1,011
2 votes
1 answer
537 views

Time Series Classification using Transformer Encoder

Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For ...
Zohaib Hamdule's user avatar
0 votes
1 answer
131 views

Embeddings/tokenizers for a transformer with binary-valued data

I'm trying to train an encoder-decoder transformer model for completion of binary-valued data. The each input is basically a length-n bitstring $x = (x_1, \dots, x_n) \in \{0,1\}^n$, generated ...
forky40's user avatar
  • 163
0 votes
1 answer
347 views

Difference between multi-head and single-head attention

Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
tolgarecep's user avatar
0 votes
0 answers
35 views

Classification in BERT - why not use class as a feature?

I am currently following this post, which details how BERT was trained. I had a few questions about the classification task: In the post, it mentions that the authors of BERT decided to add ...
Victor M's user avatar
  • 319
0 votes
1 answer
316 views

Why not use input padding in the first attention block in transformer decoder

I was studying the transformer decoder code below in Keras/Tensorflow. It was not clear how they made making decisions. In the first attention block below (self.attention_1), why did they use ...
Chika's user avatar
  • 1
0 votes
0 answers
81 views

Alternatives to classical softmax for better aggregation

I am doing machine learning on phylogenetic trees. Species within the trees have several relations, such as living at the same time or not, and I would like to use a global attention mechanism which ...
25laps's user avatar
  • 43
0 votes
0 answers
27 views

How is metadata represented in sentiment analysis?

There are papers on semantic analysis using metadata such as "Sentiment Classification on Steam Reviews" (https://cs229.stanford.edu/proj2017/final-reports/5244171.pdf) and "Detecting ...
soravoid's user avatar
0 votes
0 answers
47 views

How can I compute prediction explainability (e.g. SHAP) for a classifier trained on dense embeddings (SBERT)?

I have a multiclass classification problem (intent classification). I trained an XGBoost model on a dataset that was feature extracted using SBERT embeddings. I'm trying to compute an explainability ...
Metrician's user avatar
  • 169
0 votes
0 answers
82 views

How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
The Wanderer's user avatar
0 votes
0 answers
45 views

Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"

I am fairly new to Transformers thus this schema confuses me a lot. I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
user3187119's user avatar
1 vote
1 answer
163 views

Machine learning model for matching records

I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm ...
Björn's user avatar
  • 33.5k
2 votes
1 answer
1k views

In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?

We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as $$\alpha = Softmax(QK^T)$$ In the case of ...
Arun prakash's user avatar
1 vote
0 answers
197 views

How to determine EC2 instance type and memory for LLM inference endpoint [closed]

I am trying to estimate the costs required for hosting a fine tuned large language model for real time inference. There will be 100s of users querying the endpoint concurrently for multiple use cases ...
user3711946's user avatar
0 votes
1 answer
172 views

Why the positional embeddings for different positions are not confined

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension))) we know <...
Farhang Amaji's user avatar
1 vote
2 answers
97 views

Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?

In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
Glue's user avatar
  • 485
0 votes
1 answer
179 views

Why the positional embeddings for a specific position i, for each embedding element is different

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension))) we know <...
Farhang Amaji's user avatar
4 votes
2 answers
456 views

Necessity of keys and queries in a decoder-only transformer model

when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$. However, in the next step, we compute a dot product on the embedding ...
25laps's user avatar
  • 43
2 votes
1 answer
107 views

Why does ViTForMaskedImageModeling not construct the original image correctly?

I was trying to use masked image modeling in huggingface and I saw ViTForMaskedImageModeling in their documentation but I did not understand how it reconstructs the original image ...
floyd's user avatar
  • 1,382
1 vote
1 answer
606 views

Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?

I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part. In attention we use softmax ...
floyd's user avatar
  • 1,382
0 votes
1 answer
259 views

Effect of dK (key vector dimension) in transformers

Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers? My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
Anonymous Scientist's user avatar
1 vote
1 answer
341 views

Why the Transformer model does not require negative sampling but word2vec does?

Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side. For word2vec models, negative sampling is used for computational reasons: Is negative sampling ...
CyberPlayerOne's user avatar
2 votes
1 answer
194 views

Expectation of the absolute value of the product of correlated jointly gaussians?

I am reading the Performer paper https://arxiv.org/abs/2009.14794. To understand their ReLU kernel used to approximate softmax attention, I need to evaluate $\mathbb{E}[ReLU(x^T w) \cdot ReLU(y^T w)]$ ...
N. Menet's user avatar
4 votes
1 answer
599 views

How are the embedding split between multihead in transformers

I red the paper about transformers, and fully understood every single piece of that, so now I'm implementing it from scratch in tensroflow, without using any shipped layer from the library. The only ...
Alberto's user avatar
  • 1,217