Questions tagged [seq2seq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Use multiple softmax in transformers output layer and calculate loss

Can I use multiple softmax in the last output layer in transformers? If so, how can I calculate loss from that. I am working in pytorch. And I am asking because my data is a sequence of tuples where, ...
2
votes
0answers
35 views

Canonical LSTM backpropagation equations

I'm trying to understand the underlying mechanisms of LSTM from a programming perspective. I am no math person, and a lot of articles and papers look like alphabet soup to me. But I thought that if I ...
0
votes
0answers
9 views

How to predict integer sequence from multi feature real input of time series with lstm seq2seq

I’ve been struggling with the seq2seq problem for a while. I have multi feature input of timeseries, about 10 features(less after PCA, but it is not in main focus of the question) and I need to ...
1
vote
1answer
78 views

Does it make sense to use attention mechanism for seq-2-seq autoencoder for anomaly detection?

So I want to train LSTM sequence to sequence model, autoencoder, for anomaly detection. The idea is to train it on normal samples and when anomaly comes into model it will not be able to reconstruct ...
0
votes
1answer
26 views

Do we need to truncate test dataset for seq2seq LSTM?

I am running a summarization model which uses a seq2seq biLSTM with an attention mechanism. It is a standard practice to truncate the input dataset during training to 400 - 500 tokens. My question is, ...
0
votes
0answers
27 views

ML approach for complex Pattern-finding

I am trying to build a model that can find patterns in data (like seq2seq but for a very specific domain). So I have 2 files- the first one has all the input sequences of the same length and the ...
0
votes
0answers
12 views

seq2seq models with attention: which components are truly duplicated per word of the source sentence and which are just unrolled?

I'm reading jalammar's description of seq2seq with attention and tensorflow impl of the same. Let's say that we're trying to translate a sequence (sentence) of 4 words ...
0
votes
0answers
11 views

Inconsistent and ambiguous dimensions of matrices used in the Attention layer in GNMT or text-to-speech synthesis?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
0
votes
0answers
89 views

How exactly does conv1d filter work when operating on a sequence of characters?

I understand convolution filters when applied to an image (e.g. an 224x224 image with 3 in-channels transformed by 56 total filters of 5x5 conv to a 224x224 image with 56 out-channels). The key is ...
0
votes
1answer
34 views

Aggregating LSTM subsequence output into full sequence

I have an n->n seq2seq LSTM that takes a sequence of length n and produces a sequence of length ...
1
vote
1answer
94 views

Does the Transformer decoder query based on the previous token?

Consider the decoder part of the popular Transformer architecture; briefly put, the decoder module consists of a composition of self-attention layers and performs auto-regressive prediction. Because ...
0
votes
0answers
28 views

How is the length of the output for encoder-decoder (seq2seq) models determined

I am trying to understand how encoder-decoder models works. The encoder receives a sequence and the length is known. However the output of the encoder is just a single word vector capturing the ...
0
votes
0answers
29 views

Order-insensitive variable-length sequence encoder?

While brainstorming on a new project, I stumbled upon a problem. I would like to create a neural network to encode a sequence of objects and get the gist of the sequence. However, the sequence is of ...
2
votes
1answer
31 views

RNN Regression outputting Same(ish) values

I have a sequence to sequence LSTM (encoder/decoder model) that I made following this tutorial. I'm trying to output a series of human poses (in the form of 3D coordinates) with shape (N, 17, 3). I'm ...
2
votes
0answers
79 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
0answers
12 views

What is currently the best way to add a custom dictionary to a neural machine translator that uses the transformer architecture?

It's common to add a custom dictionary to a machine translator to ensure that terminology from a specific domain is correctly translated. For example, the term server should be translated differently ...
0
votes
1answer
238 views

Can we actually get the probability of a text using recurrent neural network?

I know that recurrent neural network is used to generate text and to model the probability of $P(x_0,x_1,x_2,x_3)=P(x_0)P(x_1|x_0)P(x_2|x_1,x_0)P(x_3|x_2,x_1,x_0)$ where $x_i$ is words/text. If RNN ...
0
votes
1answer
103 views

Transformer based decoding

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder ...
3
votes
1answer
184 views

“Attention is all you need” input scaling explanation

I would like to ask about the last sentence here from paper https://arxiv.org/abs/1706.03762: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings to ...
0
votes
1answer
97 views

Adding context in a seq2seq RNN model

The encoder of a seq2seq model is meant to generate a conditioning context for the decoder, as mentioned here A RNN layer (or stack thereof) acts as "encoder": it processes the input sequence and ...
1
vote
1answer
831 views

seq2seq in pytorch [closed]

I have an encoder LSTM whose last hidden state feeds to the decoder LSTM. If i call backward on the loss for the decoder lstm, will the gradients propagate all the way back into the encoder as well, ...
0
votes
1answer
132 views

Is it possible to use seq2Seq models to predict HTML code from XML file?

I have XML file that describes some embedded components. So the file has different markups that correspond to different fields. The intention behind this project is to generate automatically UI ...
2
votes
0answers
35 views

Sequence to sequence with real number features?

This is probably related to this question: How to make a seq2seq model work with infinite vocabulary? I have read many seq2seq implementations and they all seem to only work on fixed, well-...
1
vote
1answer
369 views

what's the difference between seq-to-seq and many-to-many?

I'm trying to build a model to forecast multi-steps ahead time series data like stock market data. And I found out that many precedents projects are done with many-to-many RNNs, not seq-to-seq. Is ...
0
votes
0answers
31 views

Differentiable remover of repetition?

In a pet project to see why automatic phonetic transcription is so hard, I would like to use both audio data where each sample is annotated by the sound to which it belongs (a phonetic segments tier) ...
3
votes
1answer
173 views

In Sequence to Sequence learning, how can large amounts of missing/special words in a sentence be compensated for?

I'm currently working on a Seq2Seq model for a chatbot and I'm converting every sentence to numerical vectors with word embeddings, i.e. GloVe. My problem is that training doesn't progress; the ...
2
votes
1answer
944 views

In seq2seq, how is the attention vector combined with the hidden state of the decoder?

My understanding of attention is that a weighted combination of a set of vectors is somehow combined with the decoder's hidden state. How exactly is it combined? Is it added to the hidden state before ...
3
votes
2answers
1k views

How to make a seq2seq model work with infinite vocabulary?

I have trained a translation seq2seq model. In my model, I have kept vocabulary size to 100,000. This constraint limits my model from generating any words which are not in this 100,000. So how does ...