Questions tagged [transformers]
Transformers are part of a deep learning model family using the attention mechanism to learn how to make prediction over sequential data without the need for recursion.
163
questions
1
vote
0
answers
9
views
How to customize the number of encoders/decoders in a pre-trained transformer [migrated]
I am implementing a pretrained transformer model using Python's transformer module to perform text summarization and I would like to compare the performance of the fine-tuned BART transformer given ...
- 11
0
votes
0
answers
11
views
What are the layer normalization dimensions in transformer?
In transformer training, the activations have three dimensions: batch, feature (i.e. embedding) and time (i.e. token). Layer normalization is applied, calculating statistics (mean, standard deviation) ...
- 1,041
0
votes
0
answers
21
views
How does chatgpt decoder identify the needed tokens for text generation?
I understand that the encoder architecture deals with encoding words into numbers and encoding the relationship among words but on the decoder part where it decides what are the certain tokens to ...
0
votes
0
answers
20
views
Why relu not used after the second FCL in the feed forward neural network part of the transformer
I found that after the second fully connected layer of feed forward neural network, transformer doesn't apply ReLU on it.
If the purpose of ReLU after the first FCL, the ReLU should be applied after ...
0
votes
0
answers
7
views
Aren't layer normalizations on ViTs acting just like batch normalization?
I was checking some ViTs implentation like this.
And, as Layernorm (LN) are assigned by channels like ln1 = nn.LayerNorm(768), and the data input of this layer has ...
- 158
0
votes
0
answers
31
views
LLMs' latency and their usability for inference
I am trying to use a transformer decoder (LLM, for simplicity) to label a collection of texts, later to be used for training a classifier.
I tried multiple 7B models, which I can save on my local ...
- 326
0
votes
0
answers
11
views
Contrast Self attention and Attention in machine learning for non technical person?
How would you contrast Self attention and Attention in machine learning for a non technical person?
I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
- 821
2
votes
1
answer
104
views
Time Series Classification using Transformer Encoder
Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For ...
0
votes
1
answer
50
views
Embeddings/tokenizers for a transformer with binary-valued data
I'm trying to train an encoder-decoder transformer model for completion of binary-valued data. The each input is basically a length-n bitstring $x = (x_1, \dots, x_n) \in \{0,1\}^n$, generated ...
- 163
0
votes
1
answer
117
views
Difference between multi-head and single-head attention
Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
- 21
0
votes
0
answers
17
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 269
0
votes
1
answer
112
views
Why not use input padding in the first attention block in transformer decoder
I was studying the transformer decoder code below in Keras/Tensorflow. It was not clear how they made making decisions.
In the first attention block below (self.attention_1), why did they use ...
- 1
0
votes
0
answers
30
views
Alternatives to classical softmax for better aggregation
I am doing machine learning on phylogenetic trees. Species within the trees have several relations, such as living at the same time or not, and I would like to use a global attention mechanism which ...
- 43
0
votes
0
answers
24
views
How is metadata represented in sentiment analysis?
There are papers on semantic analysis using metadata such as "Sentiment Classification on Steam Reviews" (https://cs229.stanford.edu/proj2017/final-reports/5244171.pdf) and "Detecting ...
- 1
0
votes
0
answers
23
views
How can I compute prediction explainability (e.g. SHAP) for a classifier trained on dense embeddings (SBERT)?
I have a multiclass classification problem (intent classification). I trained an XGBoost model on a dataset that was feature extracted using SBERT embeddings. I'm trying to compute an explainability ...
- 159
0
votes
0
answers
39
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 657
0
votes
0
answers
33
views
Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"
I am fairly new to Transformers thus this schema confuses me a lot.
I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
1
vote
1
answer
66
views
Machine learning model for matching records
I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm ...
- 30.4k
0
votes
0
answers
23
views
Black box model and white box model fusion
I have two models:
Black Box: Fedformer https://github.com/Zero-coder/FEDformer
White Box: A linear function
I want to predict home load, I use Black box predic time-series got a pth file. Then I use ...
- 11
1
vote
1
answer
573
views
In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?
We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as
$$\alpha = Softmax(QK^T)$$
In the case of ...
- 11
0
votes
0
answers
62
views
What are the similarities and the differences between MaskGit and Diffusion models
maskgit and diffusion - can you explain the similarities and differences between these two types of models?
- 3,200
1
vote
0
answers
167
views
How to determine EC2 instance type and memory for LLM inference endpoint [closed]
I am trying to estimate the costs required for hosting a fine tuned large language model for real time inference. There will be 100s of users querying the endpoint concurrently for multiple use cases ...
- 111
0
votes
1
answer
48
views
Why the positional embeddings for different positions are not confined
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
0
votes
1
answer
44
views
Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?
In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
- 443
0
votes
1
answer
92
views
Why the positional embeddings for a specific position i, for each embedding element is different
for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))
we know <...
- 169
4
votes
2
answers
201
views
Necessity of keys and queries in a decoder-only transformer model
when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$.
However, in the next step, we compute a dot product on the embedding ...
- 43
0
votes
0
answers
17
views
Is there any reference about backpropagation of the Transformer's multi-head layer?
Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
- 51
1
vote
1
answer
58
views
Why does ViTForMaskedImageModeling not construct the original image correctly?
I was trying to use masked image modeling in huggingface and I saw ViTForMaskedImageModeling in their documentation but I did not understand how it reconstructs the original image ...
- 1,360
0
votes
0
answers
64
views
How can I use the features embedding from wav2vec model?
I learned how to get the embedding features from wav2vec from this question but I have some questions.
What is the difference between using those fixed embedding features
in a downstream task and ...
- 1,360
0
votes
0
answers
319
views
Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?
I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part.
In attention we use softmax ...
- 1,360
0
votes
1
answer
54
views
Effect of dK (key vector dimension) in transformers
Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers?
My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
0
votes
1
answer
182
views
Why the Transformer model does not require negative sampling but word2vec does?
Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side.
For word2vec models, negative sampling is used for computational reasons:
Is negative sampling ...
- 2,365
2
votes
1
answer
125
views
Expectation of the absolute value of the product of correlated jointly gaussians?
I am reading the Performer paper https://arxiv.org/abs/2009.14794. To understand their ReLU kernel used to approximate softmax attention, I need to evaluate $\mathbb{E}[ReLU(x^T w) \cdot ReLU(y^T w)]$ ...
- 23
0
votes
0
answers
92
views
Create weight for tokens with BERT models
I am fine-tuning CamemBERT model for text classification.
I have a lot of domain specific words and a small dataset (10k sentences with 70 labels) and when I added tokens, it didn't help the model to ...
- 33
0
votes
0
answers
14
views
In the Transformer, how do I get the Keys and the Values from the output of the top encoder (which go into Encoder-Decoder Attention)? [duplicate]
I am trying to implement Attention is All You Need paper in PyTorch without looking at any code.
I'm struggling to understand how do I get the Keys and the Values from the output of the top encoder. ...
0
votes
0
answers
64
views
Can the same training data be used for MLM and fine-tuning of a transformer model?
Can I use the same training and validation data to perform MLM and train the weights of a classification head?
Here is the background of my specific problem:
The problem is a binary classification ...
- 31
4
votes
1
answer
249
views
How are the embedding split between multihead in transformers
I red the paper about transformers, and fully understood every single piece of that, so now I'm implementing it from scratch in tensroflow, without using any shipped layer from the library.
The only ...
- 859
0
votes
0
answers
21
views
Abnormal peaks in loss curve when training a transformer model for time series prediction [duplicate]
Model type- Encoder Decoder type Transformer time-series model.
(Based on: https://github.com/KasperGroesLudvigsen/influenza_transformer)
Training parameters:
Learning rate- ...
1
vote
1
answer
383
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
1
vote
0
answers
58
views
Explainable AI - Noise in gradients and embeddings of large language models
I am doing experiments related to explainable AI. I have two BERT models - the standard bert-base-cased and a distilled ...
- 11
2
votes
1
answer
126
views
Transformers for sales forecasting, vector output
I'm looking for a multivariable time series architecture that accepts multiple independent variables and produces multiple dependent variables. The context is sales forecasting given item, discount ...
- 2,501
2
votes
1
answer
408
views
How do LLMs transform tokens into vectors?
I know about tokenization algorithms like BPE and some other basics of tokenization from the Hugging Face course. I've also heard about word2vec and other algorithms for assigning words to vectors. I'...
- 133
1
vote
0
answers
84
views
Numerical Stability of Transformer Training
I am trying to train a Transformer for sequence model, specifically for time series denoising. I have observed that the loss function (MSE) has been significantly improved during the evaluation which ...
- 11
3
votes
1
answer
632
views
How does masking work to make neural networks use varying input lengths
A well known solution to make Neural Networks (NNs) work with varying lengths of input data is to use padding and then masking the padded inputs (padded with zero, a common approach in natural ...
- 745
0
votes
0
answers
42
views
Transformer model for statement consistency / stance detection
As an escience support programmer, I often get requests for research questions on checking a corpus for stances, or consistency with a particular statement. For example, one might wonder which news ...
- 165
0
votes
0
answers
19
views
Prediction model for [parameter vector] to [time series]
Say I have a function $F$ that takes in a parameter vector $P$ (say, a 5-element vector), and produces a (numerical) time series $Y[t]$ of length $T$ (eg $T$=100, so $t=1,...,100$). The function could ...
- 117
1
vote
1
answer
1k
views
In Transformers, for the maximum length of Encoder's input sequences and Decoder's input sequences - should they be two different numbers?
I noticed that there's usually a limit for the input length of transformers. But considering there are actually two input layers - one for the Encoders and one for the Decoders, can we and should we ...
- 2,365
0
votes
1
answer
1k
views
Difference between "transfer learning" and "fine-tuning"
I am currently reading the BERT paper and it splits the use of pre-trained models into two categories:
Feature-based whereby you just take the embeddings for the tokens and plug it into whatever ...
- 1,014
1
vote
1
answer
869
views
How does inference work on a Transformer?
(Let's say I trained a transformer for translation.)
In the training, the output sentences are given and fed into the decoder as a whole. However, with inference, only a start-of-sentence (SOS) token ...
1
vote
1
answer
209
views
Why are K and V the same in the second attention layer of a Transformer's decoder?
(For this example, let's say we're using a Transformer to translate from English to French.)
For the decoder in a Transformer, the second attention layer takes K and V from the encoder and then takes ...