Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
0
votes
0answers
31 views

Need help solving gradients for GRU Recurrent Neural Network

I am trying to code my own RNN with GRU cells. I have 12 total parameters that need to be trained and so far have successfully calculated the gradients for 8 of those parameters. I would be extremely ...
0
votes
0answers
15 views

Activation function between LSTM layers

I'm aware the LSTM cell uses both sigmoid and tanh activation functions internally, however when creating a stacked LSTM architecture does it make sense to pass their outputs through an activation ...
1
vote
1answer
22 views

What is the purpose of the update gate and how does it achieve it in a LSTM?

I understand how the forget gate works. My understanding of the forget gate: A sigmoid function is used to make each of the gate tensor's values $\Gamma_f^{<t>}$ range from 0 to 1. The forget ...
0
votes
1answer
17 views

LSTM - Who are the inputs for those hidden cells?

I'm learning RNN and I'm understanding, but I have a specific question that I can not find answer: What is the x input for the cells (pointed in yellow) for the ...
0
votes
1answer
24 views

Train a RNN with unknown vocabulary size

I'm new to deep learning and i'm trying to code a Visual Question answering network. I studied and (i think) understood how RNN and LSTM work. From what i'he understood, i need to train my network ...
2
votes
1answer
42 views

What is Time Lag in Recurrent Neural Network? Why is it a Problem?

I was reading a paper on LSTM's and this term "time lag" is coming quite frequently, but there is no any definition of it mentioned anywhere in the paper. It also states that LSTM really helps to ...
0
votes
1answer
86 views

Recurrent Neural Network - Vanishing Gradient in a network that has output at each time step

I am trying to understand the problem of vanishing gradient in RNN. However, it seems to me that this problem is not happen with a network that has output at each time step. Let's say we are trying to ...
0
votes
0answers
12 views

How to correctly deal with one-hot-encoding and multi-dimensional data in tensorflow RNN

I'm creating a binary classifier that classifiers letter sequences e.g 'BA'. Each sequence is made up of 2 letters encoded as one-hot vectors. For example, the sequence 'BA' is ...
0
votes
0answers
18 views

Use historical data in Feed Forward networks to replace RNN / LSTM [duplicate]

I am very confused. My current understanding of RNN / LSTM is the following: All a they do is include the previous element of the sequence as an input. Along with other inputs from this itteration, it ...
2
votes
1answer
10 views

How do RNNs used in Machine Translation have the right output length?

For machine translation the length of input and output sequences is mostly different. Typically considering an encoder-decoder architecture is used, how does the output come out to be the right length ...
0
votes
1answer
19 views

RNN model for predicting room temperatures

I am currently doing a project in Machine Learning where I am trying to predict the temperature of a room in future. I have a 1-year dataset of a house with 12 rooms. Data is collected at 10 min ...
1
vote
1answer
27 views

Do recurrent neural language models greedily model language probability?

Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
1
vote
0answers
47 views

How to predict future sequence data? [closed]

I am currently doing a signal processing project. However, this is very different to projects I did before and I am struggling to find a good start point to do this. My problem is as follows. I ...
1
vote
1answer
34 views

What are number of hidden layers in LSTM?

I new to LSTM. I have not understood some terms used while implementing it in tensorflow. So I have ECG data, with each event having 60 heartbeat templates with each heartbeat template having 600 data ...
0
votes
0answers
77 views

VRNN (Variational Recurrent Neural Network) code with Variable Input Lengths on Tensorflow

I have been trying to write VRNN (Variational Recurrent Neural Network: A recurrent latent variable model for sequential data (NIPS 2015)) with variable input data length. My problem is that it is ...
1
vote
0answers
21 views

Determining the degree of confidence of a Prediction

I'm handling a text multi class classification problem using RNN LSTM and model is pretty accurate (approximately 95%). But I'm interested in finding the degree of accuracy of a Prediction. Can I make ...
1
vote
0answers
188 views

Mean absolute percentage error returning NAN in PyTorch [closed]

I'm using mean absolute percentage error (MAPE) as a loss function for an RNN, however during training I start getting NaN values. I first used MAPE to calculate error between sequences of 3D ...
2
votes
1answer
18 views

RNN Regression outputting Same(ish) values

I have a sequence to sequence LSTM (encoder/decoder model) that I made following this tutorial. I'm trying to output a series of human poses (in the form of 3D coordinates) with shape (N, 17, 3). I'm ...
0
votes
1answer
19 views

neural network timeseries binary classifier: simple way to use temporal context

The problem I am working on is binary classification of a time series. To be more specific, input data corresponds to the 0.2s worth of accelerometer readings, ...
0
votes
1answer
46 views

Classification methods for univariate time series

Our team wants to develop a machine learning algorithm for classification of univariate data. Our data is a live feed from a position sensor placed in an injection molding machine. We want to be able ...
1
vote
0answers
11 views

Regression from a varying number of variable sized sequences to a single scalar target

I have a recurrent network (currently a LSTM, but I may be switching to a recursive net to better reflect certain domain specific assumptions) for regression (to a single real valued scalar) on time ...
0
votes
0answers
10 views

Neural network converges to bias in high multi-label scenario

I am trying to create a trigger word detection system by following the tutorial: https://github.com/Kulbear/deep-learning-coursera/blob/master/Sequence%20Models/Trigger%20word%20detection%20-%20v1....
1
vote
0answers
37 views

Training an LSTM “autoencoder” without a decoder?

Say you take a timeseries. Then you'd train the model to predict a future window, based on a previous window, i.e. learn the temporal dependence in data through forecasting. This would result in both ...
1
vote
0answers
16 views

Creating a feedback RNN

I will propose a trivial example to explain my question. So yes, there are many techniques I could use in post processing etc. but I would like the model to have the following ability. Say I have the ...
0
votes
2answers
35 views

Rigorous Logic of LSTM

I have read several introductory texts about LSTM online, but none of them give a rigorously mathematical explanation of what the model actually does. For instance, why does the forget gate forget any ...
0
votes
2answers
23 views

Source of vanishing/exploding gradients in RNN

Problem I am trying to understand the source of vanishing/exploding gradients in vanilla RNN. The update rule of vanilla RNN is $$ \begin{aligned} &\mathbf{a}^{\left<t\right>}=\...
0
votes
1answer
30 views

Taking derivative for RNN back propogation

I am trying to understand the derivation of backpropagation for recurrent neural networks (RNNs) from this source: https://github.com/go2carter/nn-learn/blob/master/grad-deriv-tex/rnn-grad-deriv.pdf ...
0
votes
0answers
27 views

Gaps in time-series for LSTM classification model

I am using an Long short-term memory (LSTM) recurrent neural network model to perform classification of accelerometer sensor data. The experiments (for collecting the data) were run a few months apart ...
0
votes
1answer
31 views

Is it valid to have different features in the same time series

In this website the authors points out the need to 12. Standardize the features. However, in my case, I want to lay out different features(ie. 1) time series for ...
0
votes
0answers
20 views

How to label time series data correctly for training RNN/CNN models?

My Case I want to tackle a deep learning classification task using various smartphone sensor data. I will use a self-built data acquisition app and basically walk around with the phone, adding class ...
1
vote
0answers
14 views

Is it possible to train an RNN to predict projectile motion?

Projectile motion is given by a function $y = -9.81 x^2 + ax + b$ for some parameters $a$ and $b$. I'll simply assume for $x$ values to be distanced by 1, so $x_t = t$. I can then easily generate ...
0
votes
0answers
16 views

RNN (LSTM) training on multiple time series

Regarding RNN training, We feed network a network -> point by point from the same time series (or image or smth else). When we “switch from one time series to another”, what should be done or how ...
3
votes
1answer
38 views

if an RNN was trained on $y=10x^3+5x+1$ will it work with for example $y = 2x^3 + x -5$ as well?

If we trained an RNN type of neural network (RNN, LSTM, GRU, etc) on a set of datapoints that were generated for example from a function such as $y = 10x^3 + 5x + 1$ , and then tried to use datapoints ...
0
votes
1answer
55 views

How can RNN handle variable sized inputs? [duplicate]

I came across this answer which is specific to Keras. But my question is at concept level. I am getting confused, How can RNN handle variable size inputs? The answer here says RNN can handle variable ...
0
votes
0answers
26 views

Literature / Example on Backpropagation Through Time Training of GRUs?

I am looking for a step by step explanation of BPTT Equations for GRUs. I found some for RNNs in general or LSTMs but none for GRUs. e.g. I am looking for something going into detail about Training ...
0
votes
0answers
13 views

Is there any algorithms/models to generate embedings of sequential data (other than RNN)?

I know that RNN can be used for such task. For instance facenet used rnn with triplet loss. But maybe there are some less sophisticated alternatives to try first?
0
votes
0answers
21 views

LSTM cell state update equation wrong in Deep Learning book? (Equation 10.41)

In Equation 10.41 of the Deep Learning Book, the author writes the update equation of LSTM cell internal state as : $$ s_{i}^{(t)}=f_{i}^{(t)} s_{i}^{(t-1)}+g_{i}^{(t)} \sigma\left(b_{i}+\sum_{j} U_{...
4
votes
1answer
206 views

RNN vs Kalman filter : learning the underlying dynamics?

Being recently interested in Kalman filters and Recurrent neural networks, it appears to me that the two are closely related, yet I can't find relevant enough litterature : In a Kalman filter, the ...
0
votes
1answer
52 views

Can we actually get the probability of a text using recurrent neural network?

I know that recurrent neural network is used to generate text and to model the probability of $P(x_0,x_1,x_2,x_3)=P(x_0)P(x_1|x_0)P(x_2|x_1,x_0)P(x_3|x_2,x_1,x_0)$ where $x_i$ is words/text. If RNN ...
0
votes
1answer
24 views

Examples of Real Applications for Time-series with Continuous-valued Targets and Continuous-valued Observations

Suppose that we are interested in estimating continuous-valued targets $y_t$ from continuous-valued observations $x_t$ over discrete time steps $t = \{1,2,3,\dots,T\}$. Could you give me some ...
1
vote
0answers
60 views

How to train an LSTM model on multiple single-variable time-series data?

I am quite new to the field. I am working on a problem involving time-series forecasting of single variable time-series. Data is collected from the pressure sensor on a patient in hospital. Time ...
0
votes
0answers
240 views

Multiple 1D Convolutional Neural Networks in PyTorch

I am very new to the field of deep learning and PyTorch, so pardon me if the question appear too basic. I am trying to build a framework based on the descriptive figure shown below in PyTorch. The ...
1
vote
1answer
60 views

How a softmax function can be used in multiclass classification?

I'm trying to understand how a softmax function help classify in multi-class classification. In Andrew Ng video, he shows how a simple 1 layer neural network, with 2 inputs ($x_1, x_2$) and 3 outputs ...
2
votes
1answer
111 views

How does one generate (smooth) varying size output signals with Machine Learning?

I am interested in knowing about generative methods that generate signals (e.g. images) of varying sizes. But the size generation being sort of "smooth/continuous". So for example, generating images ...
1
vote
1answer
89 views

How to design a Neural Network model that combines components of Feedforward and Recurrent features?

I want to design an end-to-end system that has components of both feedforward neural networks and recurrent neural networks. For example the data can have different components (some sequential in ...
0
votes
1answer
58 views

Can we use the folded version of RNN in training?

I always see that people teach the unfolded version of the RNN. Do they do that just for the sake of teaching because the unfolded version is simple to explain sequence learning on it? I read in an ...
0
votes
0answers
15 views

RNN architecture selection by AIC or other methods

I am using recurrent neural netword (RNN) to model a time series. However, I am not sure how should I do the cross-validation for hyper-parameter selection. For other dynamic model, I am using AIC (or ...
0
votes
0answers
23 views

Weight Matrices for RNN One-to-Many Architecture

It's my understanding that there are 3 weight matrices for RNNs - The one connected to the input, the one connected to the previous time-step activations and the one connected to the output. But for ...
1
vote
1answer
35 views

Sentence classification using an RNN, without classifying each sentence separately

Suppose you are trying to do sentence classification. That is, given a block of text with many sentences, I want to output the "class" of each sentence in order. For example, suppose there are ...
0
votes
1answer
88 views

Is there any paper about vanishing-gradients of LSTM? [duplicate]

Some web pages mentioned that LSTM causes the vanishing or exploding gradients if the sequence is too long. These are one of the pages mention the problem: https://machinelearningmastery.com/...