Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [lstm]

A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.

6
votes
0answers
2k views

Which loss function to use when training LSTM for time series?

I'm experimenting with LSTM for time series prediction. The example I'm starting with uses mean squared error for training the network. I know that other time series forecasting tools use more "...
6
votes
0answers
2k views

Variable importance in RNN or LSTM

Several method have been devised for accessing or quantifying variable importance (even if only relative to each other) in MLP neural network models: Connection weights Garson’s algorithm Partial ...
4
votes
0answers
29 views

Feature engineering for sheet music

I have a large dataset of digitized music scores that I'd like to use as input to a network. Initially, I'm looking to train networks to identify key signatures, tempo, dynamics, etc. from the raw ...
4
votes
0answers
179 views

How are the internal representations in ELMo averaged?

I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-...
4
votes
0answers
275 views

traditional state-space models and LSTMs

I am trying to understand the nature of LSTMs in relation to intuitions from traditional state-space models (e.g., Kalman filtering). The code below aims to simulate a simple univariate linear state-...
3
votes
0answers
642 views

How to train an LSTM when the sequence has imbalanced classes

I'm labelling sequences at every time step, but some labels in the dataset only occur very briefly between two much more common labels. As a result, the NN is biased towards these common labels. I can'...
3
votes
0answers
1k views

Rolling window time series training and validation in Keras

I have a conceptual question regarding the use of the rolling window approach for training and validating a recurrent neural network (LSTM or GRU) on time series data. I have daily time series data ...
3
votes
0answers
164 views

Is my weight matrix *learning* from all the steps in my LSTM?

I'm attempting to build an LSTM in Tensorflow to take in a series of amino acids (represented as Bitfields) and output a series of Torsion angles (4 numbers ranging from -1 to 1) for each amino acid ...
3
votes
0answers
588 views

When, if at all, to reset the state of an LSTM when training and when testing?

I am building an LSTM that takes in time-series financial data. My dataset is made up of IDs (each ID is a certain stock), and timestamps. For each ID at each timestamp, there are a number of features ...
3
votes
0answers
291 views

Truncated Back Propagation of LSTM with K2 > K1

Every K1 time steps we will run truncated back prop through time (BPTT) for k2 time steps. For k2 <= k1 I don't find any problems. But let's look at the case where k2 > k1, and particularly where ...
3
votes
0answers
725 views

Why is the derivative of the LSTM cell state w.r.t. to the previous cell state equal to the forget gate?

I keep seeing this online, on Quora and Machine Learning subreddits but I don't get it. Here's some basic math to show otherwise: We use this equation for the cell state: $c_t = f_t \odot c_t\__1 + ...
3
votes
0answers
201 views

Hessian-Free instead of LSTM for Recurrent Net Machine Translation

Last year, Ilya Sutskever and collaborators came out with a paper about a recurrent LSTM net that learns sequence to sequence mappings for machine translation. It's somewhat surprising that the ...
2
votes
0answers
18 views

LSTM Generates Duplicates

I'm using an LSTM to generate molecules using the Simplified Molecular Input Line Entry System (SMILES) to represent molecules. As an example, Aspirin is represented as O=C(C)Oc1ccccc1C(=O)O I have ...
2
votes
0answers
55 views

Time Series Forecasting RNN: Difference Between Masking & Excluding Rows

Suppose you have missing values in a time series E.g. : t1 x1 y1 t2 ? ? t3 x3 y3 t4 ? ? t5 x5 y5 You are trying to forecast this time series using a recurrent ...
2
votes
0answers
203 views

How to choose suitable Autoencoder (LSTM) architecture?

I am new to Autoencoders and I am a bit confused on which model to try for my situation and what is the difference between all the different models I have seen in tutorials. So, I have a set of time-...
2
votes
0answers
625 views

RNN LSTM overfitting

I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
2
votes
0answers
218 views

Awful performance of LSTM on noisy time series after stationarisation

Note. The post is quite long because I added some thought process for the sake of seeing the big picture. So grab a coffee and indulge yourself. For tldr the actual question on the bottom. I put my ...
2
votes
0answers
268 views

Vanishing Gradient on LSTM and GRU

LSTM and GRU are models that were proposed in order to solve the vanishing gradient issue. However, I have noticed that with long sequences these models also suffer from it, which makes sense. I am ...
2
votes
0answers
142 views

Training a bidirectional LSTM is unstable

I'm trying to solve timeseries classification problem. That's my model: ...
2
votes
0answers
95 views

LSTM - Learning a sinus function with linear part

I have recently build a simple LSTM-Network to predict a sinus function, which worked fine. Now I wanted to fit a sinus function containing a linear part with the same network but the results are ...
2
votes
0answers
134 views

Are there any rules of thumb for the number of hidden layer neurons in a RNN or LSTM for time series prediction?

Say that I have a univariate time series X(t) that I want to forecast using RNN/LSTM. I have 2 years of weekly sales data that is seasonal. How many hidden layers and neurons in each layer do I need ...
2
votes
0answers
35 views

Understanding Calculations in LSTMs

I’m trying understand LSTMs better: When we set the LSTM units in Keras or Tensorflow as: model.add(LSTM(256)) or ...
2
votes
0answers
123 views

LSTM good at hallucinating, useless at ground truth prediction?

I was interested in this project, so I cloned it and trained it on Moby Dick, for this challenge. The goal is to predict the next character given the past ground-truth characters. Overfitting is not ...
2
votes
0answers
2k views

LSTM for stock prices and trends prediction

I have an assignment to create a LSTM network predicting price and trend of cryptocurrencies based on stock market data from the past. The network I am using is a multilayered LSTM, where layers are ...
2
votes
0answers
1k views

How to use Keras pre-trained 'Embedding' layer?

guys! I've trained model in keras using Embedding on specific corpus of articles. I use this tutorial http://adventuresinmachinelearning.com/word2vec-keras-tutorial/ Now I want use it as layer in my ...
2
votes
0answers
102 views

How to learn timeseries with LSTM, having different orders of magnitude in the output?

I am relatively new to neural networks and LSTM networks in particular, but I already worked with other ML algorithms. I am currently trying to reproduce a physical model (based on ordinary ...
2
votes
0answers
664 views

LSTM for classification

I have a dataset which consists of $n_\text{samples}$ different measurements. Each measurement contains $n_\text{features}$ features. These features are for example ...
2
votes
0answers
661 views

Reinforcement Learning and Neural Networks with LSTM

I am working on a project training neural networks with an LSTM layer using Q-Learning. I haven't been able to achieve optimal results on my test bench problems. I believe my problem has to do with ...
2
votes
0answers
221 views

Generating real-valued time series with RNN

I'm trying to adapt Andrej Karpathy's char-rnn model (which is also described in Alex Graves's Generating Sequences With Recurrent Neural Networks ) to real-valued time series. Unfortunately I'm ...
2
votes
0answers
388 views

Deriving Gradients for a Vanilla LSTM

I've been banging my head on this for far too long. The following code should be easy to understand; can someone assist me in discovering what I've done wrong? The code passes a numerical gradient ...
2
votes
0answers
128 views

When training an RNN, what are the important factors for deciding how many unrollings / unfoldings to use?

As far as I understand many RNN:s are trained with back propagation over a sequence of $k$ datapoints. The RNN is "unrolled" for each datapoint, i.e. its output is fed into itself together with the ...
2
votes
0answers
488 views

Encoding multiple states and events as time series

I have data on a collection of entities, along with data about things that have happened to them over time. I am trying to encode this into a categorical time series for use in an LSTM neural network ...
2
votes
0answers
159 views

Measure similarity between $2$ variable length sequences using LSTM

I would like to build a model that measures the similarity between $2$ sequences that have different lengths. Input: $[a_1, a_2, \dots, a_{n-1}, a_n]$ and $[b_1, b_2, \dots, b_{m-1}, b_m]$ where $a_i$...
2
votes
0answers
719 views

What is the output of word embedding in LSTM tutorial?

I'm new to theano and found LSTM tutorial's embedding word so confusing as lack of print function. in build_model function we have: ...
2
votes
0answers
82 views

A conceptual question about LSTM-RNN

I am working on series prediction by LSTM-RNN. In the training stage, I use a random series (white noise ) as input to go through a system and get the output. LSTM is implemented to learn the ...
2
votes
0answers
204 views

Training a binary LSTM classifier with few true positives

I'm trying to solve a multilabel classification problem with n-binary LSTM classifiers. I have 17 classes in total, where multiple classes may be true for each example (e.g. news articles with ...
2
votes
0answers
295 views

Difficulty learning parameters in RNN?

I'm implementing an LSTM using the RNN package in Torch. I've been able to get very simple models to converge (like learning the relation f(x) = x), but haven't been able to get basic things like ...
1
vote
0answers
13 views

LSTM repeatedly giving the same words

I implemented a 2-layer LSTM based on the architecture described in https://panderson.me/up-down-attention/ I trained it with Cross Entropy, and for every step I fed in the correct previous word (...
1
vote
0answers
48 views
1
vote
0answers
46 views

How to apply dropout in LSTMs?

Dropout in fully connected neural networks is simpl to visualize, by just 'dropping' connections between units with some probability set by hyperparamter p. However, how dropout works in recurrent ...
1
vote
0answers
41 views

Getting over bid-ask bounce

High-frequency financial data is subject to bid-ask bounce. Description : Unlike traditional data based on just closing prices, tick data carry additional supply-and-demand information in the form of ...
1
vote
0answers
26 views

Limiting regression accuracy

I am currently using a LSTM Network to solve a regressional problem. The goal is to predict the payload mass using various time series as input data. For our intents and purposes estimating the mass ...
1
vote
0answers
56 views

Zero inflated binomial for excessive zeros

"What are some tricks for dealing with a zero inflated response variable when tackling a machine learning regression problem?" Answer : "One of the easiest and most intuitive methods is to run a ...
1
vote
0answers
159 views

Adding noise to time series data to increase training data

I am dealing with a weekly time series forecasting problem and I am currently investigating the use of an LSTM to make a multi-step forecast for a univariate time series. I actually have a ...
1
vote
0answers
76 views

How does the internal state of Tensorflow's “dtf.nn.dynamic_rnn()” change?

I am trying to do a uni-variate forecasting and I have come across this question quite a lot. This question is regarding TensorFlow's "dtf.nn.dynamic_rnn()". NOTE: When I say internal state I am ...
1
vote
0answers
111 views

Support Vector Machines VS LSTMs: How well it is justifiable to use LSTM for its Generalization properties?

The question is pretty straightforward, How well one can justify using LSTMs(Neural Networks) for text classification task in terms of "Generalization" compared to classic support vector machines(SVM) ...
1
vote
0answers
116 views

Interpreting accuracy graph for a LSTM model | Keyword Prediction

I have created a LSTM model for keyword prediction. I am using RMSE optimizer for training. I observe that the train and test accuracy decreases at first and then fluctuates without much difference in ...
1
vote
0answers
217 views

LSTM Training loss decreases and increases

I am new to LSTM and deep learning. I have 3000 reviews which I am trying to train on gensim pretrained model via word embedding. I have the following model where ...
1
vote
0answers
17 views

Best method for predicting a binary DV from multiple IVs with time series data in R or Python?

Let's say I have a dataset of 100.000 cases that contains 5 variables recorded over a period of time in long format, like this: ...
1
vote
0answers
149 views

Sequential Long-Text Classification with Recurrent and Convolutional Neural Networks

I am thinking to build a model for predicting events from news. Before I start this task I wanted to ask if someone have tried to build something like in the link(https://arxiv.org/pdf/1603.03827.pdf) ...