Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

0
votes
0answers
19 views

Physics/dynamic system simulation with Deep Learning

I am interested in modelling physical/dynamic systems with deep learning, and I think that recurrent neural networks should be a good way to model systems that can be normally represented by state-...
0
votes
0answers
7 views

Resetting states in tf.kers.layers.RNN to different batchsize [closed]

I am trying to use a stateful RNN in Tensorflow Eager (link). For my validation set, in the end I have leftover sequences which arent a full batch. To reset the state for new sequences, I normally use ...
1
vote
1answer
22 views

DNN architecture example

Could someone please give me an example (or where they have been used, if possible, link of a reference) of any NN architecture where a set of 1D CNNs are first used as feature extractor and then a ...
3
votes
2answers
122 views

Why do attention models need to choose a maximum sentence length?

I was going through the seq2seq-translation tutorial on pytorch and found the following sentence: Because there are sentences of all sizes in the training data, to actually create and train this ...
0
votes
1answer
54 views

How does SGD come in the picture for Sequence to Sequence models?

I was learning that seq2seq models (from the deeplearning.ai course) try to maximize: $$ \max_{y} P_{\theta}(y_1 \dots y_{T'} \mid x_1 \dots x_T ) $$ I learned that one way they do it is via beam ...
0
votes
0answers
28 views

LSTM Autoencoders - architecture

I am a bit confused about the structure of LSTM autoencoders, as far as I know, common way to construct vanilla autoencoders is bottleneck structure, for instance, start with 40 nodes, encode it to 30 ...
0
votes
0answers
15 views

How is inference performed in the RNN “many-to-one” architecture?

I'm looking at the diagram for the "many-to-one" architecture here and it looks like in the training phase there would be weights trained across activations between timesteps $W_{aa}$, between inputs ...
0
votes
0answers
13 views

What to use as a “stop” signal in a hierarchical LSTM model with continuous variable-length outputs

I am implementing a hierarchical sequence-to-sequence deep neural network model using long short-term memory (LSTM), where the bottom level of the hierarchy generates discrete outputs (characters from ...
0
votes
0answers
19 views

Recurrent Neural Network - Time Series prediction: How to decide on the sequence length and number of output neurons

Context I am currently working with Gated Recurrent Units for time series prediction. Anyhow, my question should apply to all kinds of such recurrent networks. In my data, I have timestamps of 15 ...
1
vote
1answer
32 views

Predict song genre using LSTM

I have a dataset of songs based on genres. For example, a song may hold {5, 2, 3} as scores set for Sentimental, Rock and Jazz. In total there are 800 songs sequentially arranged. I want to predict ...
0
votes
0answers
13 views

How do I implement masking in TensorFlow eager?

I am training a stateful RNN on variable length sequences (optional: see my previous question for more details). I padded the sequences to a fixed length with the value -1. The when batches are ...
0
votes
1answer
40 views

A model (neural network) for sets of arbitrary length [closed]

I've been searching for a model that is close to RNN (is well suited for investigating sets of arbitrary lengths) but is insensitive to order. I'm aware of bidirectional RNNs. I've also found a 'bag ...
0
votes
1answer
34 views

Given the universal approximation theorem, why are LSTM better than feed forward neural networks at certain tasks?

Per the universal approximation theorem, feed forward neural networks can approximate any function up to an arbitrary level of precision on the domain that they are trained on, given a sufficient ...
0
votes
0answers
37 views

Custom TF 2.0 training loop performing considerably worse than keras fit_generator - can't understand why

In trying to better understand tensorflow 2.0, I am trying to write a custom training loop to replicate the work of the keras fit_generator function. In my head, I have replicated the steps ...
0
votes
0answers
11 views

RNN outputs noisy predictions

I have an RNN that I've trained and I'm now using to generate new sequences. These sequences are basically discrete state time courses for K different states. The ...
0
votes
0answers
9 views

Learning a Hopfield network parametrizing a Hamiltonian vs RNN

I think of an RNN as parametrizing a vector field. Say we forget about the inputs, and instead just want to learn a non-linear state space model. To make it more concrete, perhaps we want to model a ...
0
votes
0answers
10 views

Using RNNs to segment long streams

I'm using an LSTM to segment a long streams of linear data into contiguous parts. My specific application is extracting key signature changes from a stream of notes. Specifically, I want to (for each ...
1
vote
1answer
57 views

Examples of “one to many” for RNN/LSTM

Are there any examples dealing with "one to many" kind of LSTM? Basically I am trying to build a model which takes an input vector $a$ and gives an output of $[b_1; b_2 ;b_3; b_4, \ldots; b_n]$ where ...
0
votes
0answers
48 views

LSTM - When to use sliding window in time series classification?

Say I have a tensor of data with shape (30, 16000, 38) - where each tuple element corresponds to ...
0
votes
0answers
8 views

when not using teacher forcing does backpropagation flow through input layers to output layers?

When training a Recurrent Neural Network that feeds outputs in as the next input(no teacher forcing) do gradients flow through input layers to output layers and also through the hidden states?
0
votes
0answers
33 views

Attention mechanism in LSTM model to predict next action

I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict ...
0
votes
1answer
28 views

How to make a sequence element-wise clustering with a RNN (preferable in Keras)

Non-Keras contributions are also welcome since the question is very concrete already. Imagine I have a sequence $S_i = s_0, s_1, ..., s_n$, where $s_k$ is the k-th element that represents an element ...
2
votes
0answers
18 views

LSTM Generates Duplicates

I'm using an LSTM to generate molecules using the Simplified Molecular Input Line Entry System (SMILES) to represent molecules. As an example, Aspirin is represented as O=C(C)Oc1ccccc1C(=O)O I have ...
0
votes
0answers
18 views

Why my LSTM model cannot predict waveform

Basically, I'm trying to use one waveform (above) to predict another waveform (). And Here are my model structure: ...
2
votes
2answers
100 views

How does an LSTM process sequences longer than its memory?

* Note: The premise of my question was incorrect in the first place. My question assumes that an LSTM maintains a separate set of weights for each time step in the memory it is given as a design ...
0
votes
1answer
34 views

How is Bidirectional-RNN different from vanilla RNN trained with both original & reverse copies of data?

I have several questions regarding Bi-RNN. The RNN here can be LSTM or GRU as well. (1) What is the input of Bi-RNN when making inference? For RNN, if I want to predict a $\hat{y}(t)$ for the target $...
4
votes
3answers
105 views

Binary target prediction using LSTM with sparse events in time

I have a data of patients that have multiple events happening in there medical history, I'd like to predict a target of having a specific targeted-event in the next 30 days. The data is timestamped ...
0
votes
1answer
63 views

How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself. However, it seems I have not ...
0
votes
0answers
18 views

How to model a time series problem for RNNs?

I have projects as input data, each project has a weekly progress report (hours of work completed in this week). A project can have an arbitrary duration, but let's say it's usually around 100 weeks, ...
3
votes
1answer
191 views

Cannot understand LSTM inference

I seem to have stumbled on a hole in my understanding around LSTMs. In short, I cannot understand how even a simple one is actually fed samples, upon inference time/training time. Here are the details:...
2
votes
1answer
139 views

What happens to the initial hidden state in an RNN layer?

I thought I knew how RNNs work, however, when I tried to actually implement it myself, I faced some issues. For one, how do we deal with the initial hidden state? ...
0
votes
0answers
38 views

Small output range and delayed output? Predicting sine using LSTM

I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict 0.5*sin(t + N) given 0.5*sin(t) as an exercise. I have tweaked the model, changing the output layer ...
1
vote
0answers
39 views

Should I remove the trend from timeseries when using DeepAR

I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM) I figured out from the paper that SageMaker's ...
0
votes
0answers
29 views

How to explain the decision rationale of time series data classification task using deep learning model

Is there a way to explain which part of the time series data the model is looking at in task that classifying time series data (e.g. video) by deep learning model? When deep learning model using RNN ...
2
votes
1answer
44 views

Machine learning for product names

I have a machine learning challenge I may be over thinking. I have a set of 3.5 million products (not unique, there are multiple instances of each product). Each product has a "description" from it's ...
0
votes
0answers
13 views

Is it possible to make LSTM model with 4dimension shape?

Hellow, wizards. I have time series data including sevaral days. I try to predict a grade of tomorrow, which is range from 0 to 100. And I assume that this grade depends on 3 time-series independent ...
0
votes
0answers
430 views

Number of Hidden Layer Nodes in Recurrent Neural Networks

There's already a decent discussion on how to select the right number of hidden layers and hidden nodes in a feed-forward neural network: How to choose the number of hidden layers and nodes in a ...
0
votes
0answers
124 views

Time-series classification of Kinect data using Keras

For my PhD project I recorded using Kinect and Myo 11 people performing Cardiopulmonary Resuscitation (CPR), repeatedly doing chest compressions to a manikin (one person per time). I collected in ...
2
votes
0answers
40 views

Why Create a Generator Manually for Neural Network when Keras has Built-in Generators? [closed]

So many tutorials I see have manually created generators in order to use the fit_generator() command in the model-fitting stage. However, these seem to exist already in Keras. E.g., ...
2
votes
0answers
59 views

Time Series Forecasting RNN: Difference Between Masking & Excluding Rows

Suppose you have missing values in a time series E.g. : t1 x1 y1 t2 ? ? t3 x3 y3 t4 ? ? t5 x5 y5 You are trying to forecast this time series using a recurrent ...
0
votes
1answer
38 views

How to train a RNN language model?

I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction. How to split the sentences from the dataset into input and ground truth during the ...
1
vote
1answer
132 views

What is the initial state of the tf.contrib.rnn.LSTMCell? [closed]

Does tf.contrib.rnn.LSTMCell assign itself an initial state of zeros or is it random for each batch or per complete run through (if I run the model twice will it have the same initial state both the ...
0
votes
1answer
22 views

How to understand “multimodal” RNNs for image captioning?

This paper Deep Visual-Semantic Alignments for Generating Image Descriptions on image captioning proposed a Multimodal Recurrent Neural Network architecture. From my understanding, the multimodal RNN ...
0
votes
0answers
49 views

What is the recommended maximum number of time steps for RNN or LSTM?

More time steps incurs longer training time, which above certain limit becomes impractical. What is the recommended maximum limit of the number of time steps for RNN or LSTM? I'm using a powerful ...
0
votes
1answer
31 views

image caption generator

I see two models of image caption generator online: In the above model, the first LSTM cell of decoder takes the entire image as an input. In the above model, all the LSTM cells of the decoder take ...
3
votes
0answers
37 views

How to incorporate uncertainty and noise information in training and prediction of neural networks?

I am trying to use RNNs to perform state estimation on noisy sensor data. The readings are from a GPS dataset and it provides $[longitude, latitude, n_{satellites}]$. The last column, which is the ...
0
votes
1answer
21 views

How does network structure (model complexity) affects covergence speed?

I trained Bi-GRU and HAN (Hierarchical Attention Networks) on my own datasets, and found HAN converges faster than Bi-GRU, within less number of epochs. What would be the reason for this? I guess ...
1
vote
1answer
27 views

how the long term memory works in an LTSM network

I am trying to understand how lstm networks work in particular the long term memory aspect. For example if we have a very long paragraph and at the start we have someone's name, Bob for example. Bob ...
2
votes
1answer
220 views

Selecting number of time lags for input in LSTM networks?

I know from theory that LSTM are meant to selectively capture long and short term dependencies in a sequence. I'm trying to implement LSTMs for a time series task and I notice that a lot of tutorials ...
0
votes
1answer
252 views

About RNN with variable length output vectors

I have several thousands samples with equal number of features (5000, they are time dependent) and I would like to predict of vectors with variable length. 1) I'm beginner in RNN, and I'd like to ...