Questions tagged [seq2seq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3
votes
2answers
844 views

How to make a seq2seq model work with infinite vocabulary?

I have trained a translation seq2seq model. In my model, I have kept vocabulary size to 100,000. This constraint limits my model from generating any words which are not in this 100,000. So how does ...
3
votes
1answer
82 views

“Attention is all you need” input scaling explanation

I would like to ask about the last sentence here from paper https://arxiv.org/abs/1706.03762: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned ...
2
votes
1answer
867 views

In seq2seq, how is the attention vector combined with the hidden state of the decoder?

My understanding of attention is that a weighted combination of a set of vectors is somehow combined with the decoder's hidden state. How exactly is it combined? Is it added to the hidden state before ...
2
votes
1answer
151 views

In Sequence to Sequence learning, how can large amounts of missing/special words in a sentence be compensated for?

I'm currently working on a Seq2Seq model for a chatbot and I'm converting every sentence to numerical vectors with word embeddings, i.e. GloVe. My problem is that training doesn't progress; the ...
2
votes
1answer
18 views

RNN Regression outputting Same(ish) values

I have a sequence to sequence LSTM (encoder/decoder model) that I made following this tutorial. I'm trying to output a series of human poses (in the form of 3D coordinates) with shape (N, 17, 3). I'm ...
2
votes
0answers
33 views

Sequence to sequence with real number features?

This is probably related to this question: How to make a seq2seq model work with infinite vocabulary? I have read many seq2seq implementations and they all seem to only work on fixed, well-...
1
vote
1answer
765 views

seq2seq in pytorch [closed]

I have an encoder LSTM whose last hidden state feeds to the decoder LSTM. If i call backward on the loss for the decoder lstm, will the gradients propagate all the way back into the encoder as well, ...
1
vote
1answer
220 views

what's the difference between seq-to-seq and many-to-many?

I'm trying to build a model to forecast multi-steps ahead time series data like stock market data. And I found out that many precedents projects are done with many-to-many RNNs, not seq-to-seq. Is ...
1
vote
0answers
17 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
0answers
9 views

What is currently the best way to add a custom dictionary to a neural machine translator that uses the transformer architecture?

It's common to add a custom dictionary to a machine translator to ensure that terminology from a specific domain is correctly translated. For example, the term server should be translated differently ...
0
votes
1answer
54 views

Can we actually get the probability of a text using recurrent neural network?

I know that recurrent neural network is used to generate text and to model the probability of $P(x_0,x_1,x_2,x_3)=P(x_0)P(x_1|x_0)P(x_2|x_1,x_0)P(x_3|x_2,x_1,x_0)$ where $x_i$ is words/text. If RNN ...
0
votes
1answer
111 views

Is it possible to use seq2Seq models to predict HTML code from XML file?

I have XML file that describes some embedded components. So the file has different markups that correspond to different fields. The intention behind this project is to generate automatically UI ...
0
votes
1answer
38 views

Transformer based decoding

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder ...
0
votes
1answer
43 views

Adding context in a seq2seq RNN model

The encoder of a seq2seq model is meant to generate a conditioning context for the decoder, as mentioned here A RNN layer (or stack thereof) acts as "encoder": it processes the input sequence and ...
0
votes
0answers
21 views

Order-insensitive variable-length sequence encoder?

While brainstorming on a new project, I stumbled upon a problem. I would like to create a neural network to encode a sequence of objects and get the gist of the sequence. However, the sequence is of ...
0
votes
0answers
8 views

What's the meaning of having a UNK token for out of vocabulary words during decoding?

Adding a UNK token to the vocabulary is a conventional way to handle oov works in tasks of NLP. It is totally understandable to have it for encoding, but what's the ...
0
votes
0answers
34 views

How can I interpret the result of get_weight of latent size in Seq2Seq model keras

My question is related to Seq2Seq models where we have LSTM as encoder and decoder. Imagine we have the Autoencoder alone, and we extract the weight associated ...
0
votes
0answers
14 views

Ranking candidate responses in Seq2seq model

I have a seq2seq model (already trained on some dataset). Now, I want to rank candidate responses to input text from most relevant to the least one. I tried to use hidden state of encoder and decoder ...
0
votes
0answers
48 views

Is negative Viterbi Loss possible?

So I'm training a sequence-labeling model with a BiLSTM-CRF architecture, and I am getting negative values on Viterbi Loss. Is this possible? I'm using the following formula in my code, as specified ...
0
votes
0answers
9 views

Is it okay to compare Test BLEU score between NMT models while using a slightly modified standard test sets?

I am using tst2013.en found here, as my test sets to get the Test BLEU score to compare to other previous models. However, I have to filter out some sentences that ...
0
votes
0answers
27 views

How to properly use BLEU score to compare your model to existing models?

So I am using the BLEU score metric to evaluate and compare my neural machine translation model's performance with existing models. However, I'm wondering how many criteria do I have to match with the ...
0
votes
0answers
31 views

Differentiable remover of repetition?

In a pet project to see why automatic phonetic transcription is so hard, I would like to use both audio data where each sample is annotated by the sound to which it belongs (a phonetic segments tier) ...