Questions tagged [seq2seq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Order-insensitive variable-length sequence encoder?

While brainstorming on a new project, I stumbled upon a problem. I would like to create a neural network to encode a sequence of objects and get the gist of the sequence. However, the sequence is of ...
2
votes
1answer
17 views

RNN Regression outputting Same(ish) values

I have a sequence to sequence LSTM (encoder/decoder model) that I made following this tutorial. I'm trying to output a series of human poses (in the form of 3D coordinates) with shape (N, 17, 3). I'm ...
1
vote
0answers
7 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
0
votes
0answers
8 views

What's the meaning of having a UNK token for out of vocabulary words during decoding?

Adding a UNK token to the vocabulary is a conventional way to handle oov works in tasks of NLP. It is totally understandable to have it for encoding, but what's the ...
1
vote
0answers
9 views

What is currently the best way to add a custom dictionary to a neural machine translator that uses the transformer architecture?

It's common to add a custom dictionary to a machine translator to ensure that terminology from a specific domain is correctly translated. For example, the term server should be translated differently ...
0
votes
1answer
42 views

Can we actually get the probability of a text using recurrent neural network?

I know that recurrent neural network is used to generate text and to model the probability of $P(x_0,x_1,x_2,x_3)=P(x_0)P(x_1|x_0)P(x_2|x_1,x_0)P(x_3|x_2,x_1,x_0)$ where $x_i$ is words/text. If RNN ...
0
votes
0answers
28 views

How can I interpret the result of get_weight of latent size in Seq2Seq model keras

My question is related to Seq2Seq models where we have LSTM as encoder and decoder. Imagine we have the Autoencoder alone, and we extract the weight associated ...
0
votes
0answers
13 views

Ranking candidate responses in Seq2seq model

I have a seq2seq model (already trained on some dataset). Now, I want to rank candidate responses to input text from most relevant to the least one. I tried to use hidden state of encoder and decoder ...
0
votes
0answers
34 views

Is negative Viterbi Loss possible?

So I'm training a sequence-labeling model with a BiLSTM-CRF architecture, and I am getting negative values on Viterbi Loss. Is this possible? I'm using the following formula in my code, as specified ...
0
votes
1answer
33 views

Transformer based decoding

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder ...
0
votes
0answers
9 views

Is it okay to compare Test BLEU score between NMT models while using a slightly modified standard test sets?

I am using tst2013.en found here, as my test sets to get the Test BLEU score to compare to other previous models. However, I have to filter out some sentences that ...
0
votes
0answers
23 views

How to properly use BLEU score to compare your model to existing models?

So I am using the BLEU score metric to evaluate and compare my neural machine translation model's performance with existing models. However, I'm wondering how many criteria do I have to match with the ...
2
votes
0answers
56 views

“Attention is all you need” input scaling explanation

I would like to ask about the last sentence here from paper https://arxiv.org/abs/1706.03762: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned ...
0
votes
1answer
37 views

Adding context in a seq2seq RNN model

The encoder of a seq2seq model is meant to generate a conditioning context for the decoder, as mentioned here A RNN layer (or stack thereof) acts as "encoder": it processes the input sequence and ...
0
votes
0answers
31 views

What is the best model for keyphrase extraction from super long text?

I’m working on a keyphrase extraction task. The biggest difficulty of this task is that the text is very long (5000-20000 words). I’ve tried several unsupervised algorithms such as Tf-idf and TextRank ...
1
vote
1answer
756 views

seq2seq in pytorch [closed]

I have an encoder LSTM whose last hidden state feeds to the decoder LSTM. If i call backward on the loss for the decoder lstm, will the gradients propagate all the way back into the encoder as well, ...
0
votes
1answer
108 views

Is it possible to use seq2Seq models to predict HTML code from XML file?

I have XML file that describes some embedded components. So the file has different markups that correspond to different fields. The intention behind this project is to generate automatically UI ...
2
votes
0answers
33 views

Sequence to sequence with real number features?

This is probably related to this question: How to make a seq2seq model work with infinite vocabulary? I have read many seq2seq implementations and they all seem to only work on fixed, well-...
1
vote
1answer
196 views

what's the difference between seq-to-seq and many-to-many?

I'm trying to build a model to forecast multi-steps ahead time series data like stock market data. And I found out that many precedents projects are done with many-to-many RNNs, not seq-to-seq. Is ...
0
votes
0answers
30 views

Differentiable remover of repetition?

In a pet project to see why automatic phonetic transcription is so hard, I would like to use both audio data where each sample is annotated by the sound to which it belongs (a phonetic segments tier) ...
2
votes
1answer
148 views

In Sequence to Sequence learning, how can large amounts of missing/special words in a sentence be compensated for?

I'm currently working on a Seq2Seq model for a chatbot and I'm converting every sentence to numerical vectors with word embeddings, i.e. GloVe. My problem is that training doesn't progress; the ...
2
votes
1answer
847 views

In seq2seq, how is the attention vector combined with the hidden state of the decoder?

My understanding of attention is that a weighted combination of a set of vectors is somehow combined with the decoder's hidden state. How exactly is it combined? Is it added to the hidden state before ...
3
votes
2answers
826 views

How to make a seq2seq model work with infinite vocabulary?

I have trained a translation seq2seq model. In my model, I have kept vocabulary size to 100,000. This constraint limits my model from generating any words which are not in this 100,000. So how does ...