Questions tagged [transformers]

Transformers are part of a deep learning model family using the attention mechanism to learn how to make prediction over sequential data without the need for recursion.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

How truncation works when applying BERT tokenizer on the batch of sentence pairs in HuggingFace? [closed]

Say, I have three sample sentences: ...
user avatar
1 vote
1 answer
13 views

In a tranformer, the same word can have different attention weights in different sentences?

I'm trying to understand the transformer architecture for NLP. The main issue is regarding the attention weights. The same word can have different attention weights in different sentences, right?
user avatar
  • 410
0 votes
0 answers
13 views

How to properly mask MultiHeadAttention for sliding window time series data

I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
user avatar
2 votes
1 answer
22 views

BERT MLM - 80% [MASK], 10% random words and 10% same word - how does this work?

I have noticed that (from the original BERT paper) in the MLM training procedure, the authors decide to mask 15% of the words in a sentence. The mask works as following: The masked words are ...
user avatar
1 vote
1 answer
34 views

How to obtain Key, Value and Query in Attention and Multi-Head-Attention

I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
user avatar
  • 121
0 votes
0 answers
11 views

GPT-2: Why should text classification work on the last output embedding?

When GPT-2 is fine-tuned for text classification (positive vs. negative), the head of the model is a linear layer that takes the LAST output embedding and outputs 2 class logits. I still can't grasp ...
user avatar
0 votes
0 answers
12 views

Updating temporal embeddings depending on the input

I'm building a forecasting model and I'm using a temporal embedding along with a positional embedding following the same architecture as Informer. ( https://arxiv.org/abs/2012.07436 ) My problem is ...
user avatar
0 votes
0 answers
9 views

Terminology clarification "decoder" (as in encoder-decoder model)

In an encoder-decoder model such as the Transformer model, what parts make exactly the decoder? Specifically, for designing a software implementation, and I want to implement a ...
user avatar
  • 1,155
0 votes
0 answers
12 views

Is the recursive form in a transformer efficient for long sequences?

I am reading the paper Transformers are RNNs:Fast Autoregressive Transformers with Linear Attention, where the authors propose the linear transformer, which seems to be very efficient in practice. Yet,...
user avatar
0 votes
1 answer
19 views

How to understand the relations in matrix multiplications in deep learning?

In the scaled dot product attention they multiply a "softmaxed" matrix (which has shape (sequence_length, sequence_length) I think?) to the V matrix as shown What does the second purple ...
user avatar
0 votes
1 answer
45 views

Positional Encoding and Fourier transforms

While I was studying Positional Encoding, I came across an article that links coding resolution to Fourier transforms: "For anyone who has studied finite Fourier transforms, this problem should ...
user avatar
0 votes
0 answers
50 views

Using BERT Embeddings + Standard ML for text classification

I am trying to automatically detect whether a text is written by a Machine or a Human. My first approach was using a TF-IDF to build features for a logistic regression classifier, where I got an ...
user avatar
0 votes
0 answers
7 views

Suggestion regarding usage of pre-trained BERT

I have recently been working with the pre-trained BERT. It produces quite good results on supervised tasks with just a bit fine tuning. But now I wonder if I want to perform some unsupervised task on ...
user avatar
0 votes
0 answers
38 views

How to handle problem of different random seeds giving drastically different test scores in machine learning model?

For a rigorous empirical analysis, I am training a model with three different seeds - 0, 1 and 2. In each case, I found that the model obtained through early stopping (lowest validation loss) had an ...
user avatar
2 votes
1 answer
61 views

Loss function for vectors when magnitude of elements and their position are both important

Context: I am using a transformer for time series prediction. The target and predicted tensors are both of size (8, 10, 181) which represents (batch_size, number of predictions, no. of elements in ...
user avatar
  • 21
2 votes
2 answers
133 views

Does dropout have any benefits when overfitting isn't a concern?

I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training ...
user avatar
  • 121
1 vote
0 answers
82 views

Multi-Head Attention in ViT

I need help to understand the multihead attention in ViT. Here's the code I found from GitHub: ...
user avatar
0 votes
1 answer
42 views

Trainable weights in attention mechanism

I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
user avatar
6 votes
1 answer
792 views

Why are residual connections needed in transformer architectures?

Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training. This problem is ...
user avatar
0 votes
0 answers
15 views

What's the role of masking in transformer and BERT?

I've recently implemented the architectures of Transformer and BERT and found that they both have common property - masked layers among one of them. I have come to questions like below. As far as I ...
user avatar
  • 111
0 votes
0 answers
9 views

Transformer not training correctly

I'm trying to train a Transformer neural network. I am using an this implementation (although my problem is likely independent of the implementation). The problem: the loss measured by ...
user avatar
  • 11
2 votes
0 answers
80 views

Formula to compute approximate memory requirements of Transformer models

I would like to roughly estimate the memory requirement of training an arbitrary Transformer model $M$, with $l$ layers, $h$ attention heads, an embedding dimension of $d$, and an input dimension of $...
user avatar
  • 121
1 vote
0 answers
19 views

Softmax computation in Transformers

In word embedding model word2vec, computation of softmax is expensive process and hence we use many alternative as provided here. Prominently, ...
user avatar
  • 101
0 votes
0 answers
15 views

Bottleneck Layer VAE - values centred around 0 with small standard deviations

As the title states, I'm curious as possible theoretical explanations for why might VAE bottleneck layer have values more or less centred around 0 with small standard deviations. I have found this out ...
user avatar
1 vote
0 answers
29 views

Training loss, validation loss and WER decrease, then increase [duplicate]

I am trying to use Hugginface Datasets for speech recognition using transformers using this tutorial, epochs=30, steps=400, train_batch_size=16. Training loss, validation loss and WER decrease, and ...
user avatar
1 vote
0 answers
78 views

Variable batch size for inputs of different length

We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
user avatar
0 votes
0 answers
14 views

Multi-decoders may destroy VGG19 model property

A question about some statement that I didn't understood from the paper about stylization-based image colorization Stylization-Based Architecture for Fast Deep Exemplar Colorization (DOI: 10.1109/...
user avatar
0 votes
0 answers
5 views

How to apply loss to masked sequence modelling

Suppose I have the following input features and target label: ...
user avatar
  • 509
0 votes
0 answers
6 views

Make output of a sequential model self-consistent

I'm training a sequential model on the following type of sequence (just showing the target labels here): ...
user avatar
  • 509
1 vote
0 answers
16 views

Why do transformers use separate K and V networks

In transformers, two separate networks $\phi_k, \phi_q$ compute the keys and queries. The attention weight is then the dot product of the keys and queries. (i.e. $score = \phi_k(z_1) \cdot \phi_q(z_2)$...
user avatar
0 votes
0 answers
15 views

An efficient way to encode & embed tabular data of a video into a transformer?

So a little bit of a background: I have a folder which contains video files of lets say humans doing a certain action (i.e. walking) where each .2 seconds is documented in a ...
user avatar
  • 111
0 votes
0 answers
26 views

How are the self-attention layer weigths updated?

I am trying to figure out how the updating of the weights in the self attention layer works. I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
user avatar
  • 121
0 votes
0 answers
24 views

Long document summarization

I am working on Long document summarization. The models provided now cannot deal with long documents ( more than 512 or 1024 tokens). So, I am asking for some references that can help me with this ...
user avatar
0 votes
1 answer
265 views

Literature to understand the components of Temporal Fusion Transformer

I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3 However, I had to stop at page 7/8, which ...
user avatar
1 vote
0 answers
31 views

CS224(2021) self study question about transformer on random keys

I am self studying the famous Stanford class CS224 about NLP. I am struggling to make a "formal" argument with a question involving multivariate gaussian distribution even though I get ...
user avatar
  • 75
0 votes
0 answers
31 views

Issue related to Transformer decoder druing Inference using all previous output tokens until each decoder time step

I've been trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks) and understand there is a difference in the way decoder runs during training versus ...
user avatar
  • 309
0 votes
0 answers
111 views

Should Roberta be a lot better than BERT for binary text classification?

I have a binary TC problem, with about 10k short samples, and balanced class ratio. I am using pretrained BERT and Roberta for classification. With Roberta, I get 20% better results than BERT, almost ...
user avatar
0 votes
0 answers
8 views

Why do faster (eg sparse) versions of Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $ softmax(\frac{Q K^T}{\sqrt{d}}) V $, As I understand it, the runtime, assuming $\{Q, K,...
user avatar
1 vote
1 answer
1k views

What is the purpose of Add & Norm layers in Transformers?

According to the article Attention Is All You Need, each sublayer in each one of the encoder and decoder layers is wrapped by a residual connection followed by layer normalization ("Add & ...
user avatar
  • 123
1 vote
1 answer
92 views

Look ahead bias in the skip connection of the Transformer-decoder/GPT2 architecture

How come the residual connection on the attention module in Decoder-Transformers/GPT2 does not cause a look ahead bias? This is my current understanding: GPT is similar to the decoder side of the ...
user avatar
1 vote
0 answers
15 views

In attention models, why is dropout used for positional encodings?

In the "Attention is all you need" paper, they write: we apply dropout to the sums of the embeddings and the positional encodings I can understand why you might use dropout on the ...
user avatar
0 votes
0 answers
41 views

Bidirectional transformer encoder + decoder?

I want to know if there is a transformer-based encoder + decoder model architecture that can take input sequence A -> B and also take sequence B -> A? I want to perform some back-translation ...
user avatar
0 votes
0 answers
17 views

How to improve transformer performance by modifying model structure?

I am new to transformers but want to know how to improve their performance by modifying model structure. For example, a TransformerBlock adapted from text ...
user avatar
0 votes
2 answers
46 views

Why is the NSP task in BERT inconsistent or ineffective?

The NSP task is one of the two tasks in BERT which has been revolutionizing NLP, but many pretrained models abandoned that task, for instance First, XLNet removed NSP XLNet-Large does not use the ...
user avatar
  • 5,151
3 votes
1 answer
47 views

Operation modes in neural turing machine (Graves, 2014)

I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out. More ...
user avatar
  • 75
0 votes
0 answers
208 views

Transformer model training takes longer and results in lower train and validation loss

I have been making tests with Transformer model provided on Keras.io page, training for classification and seq2seq tasks in several datasets and compare Transformer to GRU/LSTM with almost same number ...
user avatar
0 votes
1 answer
74 views

Number of weights to be learnt in the encoder decoder attention in the transformer model

I have a doubt about the number of weights to be learned in the encoder-decoder attention layer in the transformer model (attention is all you need). Some blogs articles say the $K$ and $V$ (key and ...
user avatar
0 votes
1 answer
36 views

What is purpose of multihead in transformers

Why in the attention mechanism do they apply multihead? It's been said that it would lift the amplitude of vector dot production from word that at the moment is analysed, which can be correct, but ...
user avatar
0 votes
0 answers
77 views

Why transformer in deep learning is called transformer?

Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
user avatar
1 vote
1 answer
301 views

Intuition about the application of padding masks and look-ahead masks in Transformer's encoder/decoder

From the Tensorflow tutorial, the shape of the padding mask is (batch_size, 1, 1, seq_len) and look-ahead mask is ...
user avatar
  • 873