Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

2
votes
0answers
13 views

LSTM : multi-step multidimensional multivariate multi-site timeseries forecasting [on hold]

I'm working on a project in which i'm trying to do a pollution forecasting. I googled around and found that LSTM is a good candidate for this task, however, I'm still struggling at how to adapt it to ...
2
votes
1answer
34 views

Forecasting algorithms for incomplete time series data [duplicate]

I want to forecast the demand of each SKU in my warehouse every week from the history transaction that I have collected. The data contains brand, product type, SKU, quantity, date(per day), price. But ...
0
votes
0answers
18 views

Using LSTMs for Time series prediction for quantities varying with different time units

I am trying to implement neural networks - LSTMs - to implement a Time Series concept. I have 3 years of hourly data of production and I plan to use 2 years of data to train the model and predict for ...
0
votes
0answers
13 views
0
votes
0answers
10 views

Input shape for LSTM to predict customer usage quantity for a given timestep

As per my understanding the Input shape for RNN must be (Number of samples, number of timesteps, number of features). My data has 12 timesteps for each customer and number of features is 15, my ...
0
votes
0answers
26 views

State prediction of how long someone sleeps using neural nets

I have over hundred thousands of datapoints on how long individual people sleep. I also have information about how soft their beds are, their income, stress levels etc. At first I want to predicted ...
0
votes
1answer
21 views

How are RNNs with inputs greater than the defined sequence length implamented

To clarify the slightly ambiguous language in the title. I have an RNN (actually 2 stacked RNN layers) that take input X of size X [batch_size, sequence_length, features] the model is trying to ...
0
votes
0answers
4 views

RNN for predicting binary output: detection of anticipated default on panel data [on hold]

I'm working with panel data trying to detect default from a binary output (Y=1 if default) some months before (let's say 3). So I compute Y as the binary variable that is 1 if there is a default 3 ...
0
votes
1answer
41 views

LSTM output dimensionality

I am new to LSTMs. When reading the papers and websites about LSTM architecture, there is something I do not get. As I understand it, a single LSTM layer can have multiple LSTM cells (just like a ...
0
votes
1answer
46 views

Is this a sign of (bad) local minima encountered by this RNN?

On a binary classification task, I can get perfect performance on the training set, but no matter how strongly I regularize the recurrent neural network (dropout 0.99, L2 weight penalties of 0.01) the ...
1
vote
0answers
25 views

How to apply dropout in LSTMs?

Dropout in fully connected neural networks is simpl to visualize, by just 'dropping' connections between units with some probability set by hyperparamter p. However, how dropout works in recurrent ...
0
votes
0answers
28 views

Use of variable 'states' in a recurrent neural network in tensorflow? [duplicate]

The output of RNN implementations in Tensorflow is a tuple [outputs, states]. I am a bit confused about the use of the states variable. For example in this example of a simple LSTM implementation in ...
0
votes
0answers
11 views

Priming GRU in RNN: Are they doing it in the wrong way?

I am looking at some code (in PyTorch but the question is general) where they use a technique called "priming" in order to "start" the prediction of an RNN that mainly just consists of a single GRU (...
0
votes
1answer
69 views

Time steps in Keras LSTM

My understanding of time-series LSTM training is that the recurrent cell gets unrolled to a specified length (num_steps), and parameter updates are back-propagated ...
0
votes
0answers
12 views

Can attention be implemented without encoder / decoder?

I just got into models beyond biLSTM, would like to start with applying attention to my existing network (RNN). I find examples for attention always with encoder decoder architecture, however is it ...
0
votes
0answers
8 views

Demand forecasting for sequence data and multi-task learning

I have a demand forecasting problem that I'd like to solve with a deep learning using multi-task learning and I'd like advice in some areas. Problem definition: I have a set of $N$ customers that ...
0
votes
0answers
6 views

Bring improvements inside a neural network configuration

Currently, I found the right recipe for a time series regression problem to finally get acceptable to good results. Here is the config file ...
0
votes
1answer
47 views

How to deal with really sparse time series data for a binary classification task using RNN or LSTM?

I have a binary classification prediction task and more often than not, the time series data is like really sparse. The number of zeroes in the time series data is almost always more than 99%. I ...
1
vote
0answers
43 views

A mistake in Tensorflow's documation?

Tensorflow's documentation gives an example for text generation using a RNN with eager execution. To the best of my understanding, this examples defines a simple RNN (with a GRU cell and a projection ...
1
vote
0answers
15 views

How does the internal state of Tensorflow's “dtf.nn.dynamic_rnn()” change?

I am trying to do a uni-variate forecasting and I have come across this question quite a lot. This question is regarding TensorFlow's "dtf.nn.dynamic_rnn()". NOTE: When I say internal state I am ...
0
votes
0answers
5 views

Using character level model to notes generation

I would like to use character based model for music generation. Predict note after N notes. (like character model based text generators). Do to this I would like to use RNN with LSTM in Keras. I ...
1
vote
1answer
32 views

How to minimize sharpe ratio with LSTM recurrent neural network?

I've read some articles about trading using recurrent reinforcement learning such as this one. The point where I do not fully understand is how to construct the cost/loss function. In the article, ...
0
votes
0answers
26 views

Can someone explain RNN weight update derivation?

I am going through this tutorial and not able to understand derivation for weight update. Specifically I am struggling to understand the derivative of vector w.r.t matrix, as according to my knowledge ...
1
vote
1answer
36 views

Variable importance (?) for multivariate time series anomaly detection methods

I'm working on anomaly detection methods for multivariate time series $[\mathbf{x}^{new}_1,\dots,\mathbf{x}^{new}_T]$ where $\mathbf{x}^{new}_{i}$ is $p-$dimensional. I won't go into the details of ...
2
votes
1answer
28 views

Are RNNs inherently flawed? Supervised Learning assumes IID data but sequential data is not IID

From what I understand, Supervised Learning operates under the assumption that the data is I.I.D. It seems to me that the training procedure for RNNs is flawed. We receive observations in a sequential ...
1
vote
2answers
42 views

In Recurrent NN, what's the reason for adding instead of multiplying the input term and the state term in the hidden units?

As we know, the hidden layer unit has the following activation: $$h_t=tanh(UX_t+Wh_{t-1})$$ So there is the interaction between the input and the previous state: $UX_t+Wh_{t-1}$. My question is why it ...
1
vote
2answers
49 views

Mathematical structure of SimpleRNN in keras

Two types of RNN can be used: Type1: The output is being used as state h(t) = g(W1.x(t) + W2.h(t-1) + b1) Type2: There is a state in addition to the output a(t) = g(W1.x(t) + W2.a(t-1) + b1) h(t) ...
0
votes
1answer
23 views

Testing an LSTM making predictions 1 timestep into the future

Say I have a time series data set of 100 sequential timesteps, and I want to train and test an LSTM on the data set, but only on forecasting a single timestep into the future. I want more than one ...
1
vote
1answer
38 views

Can Neural Network take multidimensional inputs?

I recently learned RNN and find that a common feature of it and CNN is that they use either a LSTM cell or Convolution to process a single multidimensional input (like images and word embeddings which ...
1
vote
0answers
18 views

Interpreting accuracy graph for a LSTM model | Keyword Prediction

I have created a LSTM model for keyword prediction. I am using RMSE optimizer for training. I observe that the train and test accuracy decreases at first and then fluctuates without much difference in ...
0
votes
0answers
40 views

NaN output from Bidirectional LSTM in Keras

I am creating a bi-directional LSTM using tf.keras APIs: input_layer = tf.reshape(emb_seqs, [const.TRN_BATCH_SIZE, -1, const.VECTOR_SIZE]) lstm_layer = tf.keras.layers.LSTM(units=LSTM_UNITS, ...
1
vote
0answers
43 views

How does LSTM learn [duplicate]

I am confused on how LSTM learn from word embedding. I know that LSTM accepts 3D input (sample, timesteps, features). So, when we use embedding layer (word2vec) and we have 300-d vector representation,...
0
votes
1answer
60 views

What kind of a generative model is an RNN?

Given the taxonomy of generative models as presented by Ian Goodfellow in Tutorial on Generative Adversarial Networks (https://arxiv.org/abs/1701.00160), in what branch do we put the family of RNNs?
0
votes
1answer
9 views

Clarification about RNN/LSTM Sequence Models with Word Vector inputs

Say that we are trying to train a language model with an RNN/LSTM i.e. the inputs are words in a sentence and the outputs are the same words shifted by one such that for each input word the output is ...
0
votes
1answer
24 views

Time Series Analysis and Classification

I need to classify data that changes over time. A good example is stock prices. My problem is that stock prices examples are trying to predict the next price and I need to classify the data. I looked ...
0
votes
0answers
24 views

Using an LSTM unrolled into multiple timesteps for training, and just a single timestep for inference

I have built an LSTM in TensorFlow that is unrolled into n timesteps and trains successfully on sequences of lengths n. I would now like to use my trained model to make m daily predictions one by one....
0
votes
0answers
22 views

How to calculate negative log-likelihoog on MNIST dataset? [duplicate]

This table is from Professor Forcing: A New Algorithm for Training Recurrent Networks paper. I couldn't find their code to calculate NLL. I would like to ask if it is simply the binary cross-entropy. ...
0
votes
0answers
5 views

Find association rules between events and a failure

I am dealing with machine that provide logs of its current activity. These logs have over 4000 different messages (Info,Warning,Error,...). I would like to predict a failure of that machine using ...
0
votes
1answer
81 views

Implementing RNN policy gradient in pytorch

I want to train a recurrent policy gradient which predicts action probabilities based on prior environment states. However, I am unable to backpropagate during the "update policy" step, in which the ...
0
votes
0answers
17 views

Multiple output NN vs multiple single output NN [duplicate]

I'm trying to build a 5 input-5 output model using LSTM, where all the outputs are the same features as the inputs, predicted in the future. My question is: is it better to build 5 models, each with ...
1
vote
0answers
22 views

Is there any Good comprehensive reference for tensor calculus? [duplicate]

I am currently reading this RNN blog, where it talks about Backprop through time. I am struggling to derive it and don't understand how to go about such derivations in general. Stuff like this ends ...
0
votes
0answers
17 views

At what time step does all information become available to an input in a BRNN?

If a bidirectional RNN (BRNN) is essentially two independent RNNs put together, with the input sequence being fed in normal time order for one and in reverse time order for the other, wouldn't some of ...
1
vote
1answer
67 views

Feedforward Nets, RNNs, and LSTMs Theory

It is my understanding that feed-forward neural nets learn by backpropegating the error from comparing an outcome to the ground truth. How is this process not 'recurrent', as in recurrent nets, data ...
0
votes
1answer
25 views

Meaning of different softmax notations in papers

I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input ...
1
vote
1answer
18 views

Where is the recurrent neural network in the preprocessing layer of the “A Compare-Aggregate Model for Matching Text Sequences” model?

I read on the "A Compare-Aggregate Model for Matching Text Sequences" model that the "preprocessing layer uses a recurrent neural network": Our preprocessing layer uses a recurrent neural network ...
0
votes
0answers
68 views

Advantage of RNN over CNN/other architectures for multivariate time series prediction?

I am building a multivariate time series prediction. I want to train and use the network with fixed length series of n events. I know that I could use a RNN for this. What I do not understand though ...
0
votes
0answers
27 views

Sequential Long-Text Classification with Recurrent and Convolutional Neural Networks

I am thinking to build a model for predicting events from news. Before I start this task I wanted to ask if someone have tried to build something like in the link(https://arxiv.org/pdf/1603.03827.pdf) ...
2
votes
0answers
194 views

RNN LSTM overfitting

I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
0
votes
1answer
37 views

Preparing test data for sentiment analysis in Tensor Flow

1) I want to do Sentiment Analysis using RNN + Tensor Flow + (Keras) 2) Is it necessary to prepare test data for sentiment analysis, using RNN, (or any Neural Network), in a certain format ? If so is ...
0
votes
0answers
6 views

How to pass MDLSTM activations to a feedforward layer?

I am currently trying to implement this paper by Alex Graves on Arabic handwriting recognition. In the architecture presented in Figure 4 on page 11, it says that the image is passed to 4X2 MDLSTM, i....