All Questions
Tagged with machine-learning transformers
54 questions
0
votes
0
answers
26
views
how to classify a part of sentence using transformer
The problem is like this: I have many sentences and for each sentence I have a continuous sequence of words that I want to mask out. Then I want to train a model to classify the masked part to 1/0.
...
- 133
1
vote
1
answer
729
views
How does a transformer network's attention mechanism determine what to pay attention to?
The paper Attention is All You Need describes the transformer network which uses a "multi-head attention mechanism". The basic intuition behind this mechanism is that it weighs other tokens ...
- 9,001
13
votes
1
answer
3k
views
When calculating self-attention for Transformer ML architectures, why do we need both a key and a query weight matrix?
I'm trying to understand the math behind Transformers, specifically self-attention. This link, and many others, gives the formula to compute the output vectors from the input embeddings as:
$$Q=XW_Q,\;...
- 26.5k
2
votes
0
answers
358
views
K-fold Cross validation in Transformer-like model with epoch
I'm new to Transformer and am building a Transformer model for classification. Before I added the k-fold validation, I needed 3 epochs to train the model.
Here comes the question: say, if I use 5-fold ...
- 121