Questions tagged [lstm]

A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.

211 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
2k views

Which loss function to use when training LSTM for time series?

I'm experimenting with LSTM for time series prediction. The example I'm starting with uses mean squared error for training the network. I know that other time series forecasting tools use more "...
6
votes
0answers
2k views

Variable importance in RNN or LSTM

Several method have been devised for accessing or quantifying variable importance (even if only relative to each other) in MLP neural network models: Connection weights Garson’s algorithm Partial ...
5
votes
0answers
36 views

Feature engineering for sheet music

I have a large dataset of digitized music scores that I'd like to use as input to a network. Initially, I'm looking to train networks to identify key signatures, tempo, dynamics, etc. from the raw ...
5
votes
2answers
574 views

Which is the Recommended variants of neural network for tree like data structures?

Consider I have a training set where each data sample is a tree and each node has(including leaf node) its own feature vector. For example, single data sample will look this Now I have to classify <...
5
votes
2answers
497 views

Does LSTM Eliminate Need for Input Lags?

Does LSTM eliminate the need for input lags? I believe the answer is yes; however, I've not found it explicitly stated in the papers and searching I have completed.
4
votes
0answers
187 views

How are the internal representations in ELMo averaged?

I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-...
4
votes
0answers
336 views

traditional state-space models and LSTMs

I am trying to understand the nature of LSTMs in relation to intuitions from traditional state-space models (e.g., Kalman filtering). The code below aims to simulate a simple univariate linear state-...
4
votes
0answers
870 views

How to train an LSTM when the sequence has imbalanced classes

I'm labelling sequences at every time step, but some labels in the dataset only occur very briefly between two much more common labels. As a result, the NN is biased towards these common labels. I can'...
3
votes
0answers
1k views

Rolling window time series training and validation in Keras

I have a conceptual question regarding the use of the rolling window approach for training and validating a recurrent neural network (LSTM or GRU) on time series data. I have daily time series data ...
3
votes
0answers
170 views

Is my weight matrix *learning* from all the steps in my LSTM?

I'm attempting to build an LSTM in Tensorflow to take in a series of amino acids (represented as Bitfields) and output a series of Torsion angles (4 numbers ranging from -1 to 1) for each amino acid ...
3
votes
0answers
657 views

When, if at all, to reset the state of an LSTM when training and when testing?

I am building an LSTM that takes in time-series financial data. My dataset is made up of IDs (each ID is a certain stock), and timestamps. For each ID at each timestamp, there are a number of features ...
3
votes
1answer
1k views

Sequence lengths in LSTM / BiLSTMs and overfitting

I'm currently working with LSTMs and BiLSTMs, using Keras as library (TF backend). Following Tutorials and reading some papers, I found out that the sequences used are mostly quite short. What I do ...
3
votes
0answers
325 views

Truncated Back Propagation of LSTM with K2 > K1

Every K1 time steps we will run truncated back prop through time (BPTT) for k2 time steps. For k2 <= k1 I don't find any problems. But let's look at the case where k2 > k1, and particularly where ...
3
votes
0answers
776 views

Why is the derivative of the LSTM cell state w.r.t. to the previous cell state equal to the forget gate?

I keep seeing this online, on Quora and Machine Learning subreddits but I don't get it. Here's some basic math to show otherwise: We use this equation for the cell state: $c_t = f_t \odot c_t\__1 + ...
3
votes
1answer
479 views

Mini-batching dependent data

I have a neural network I am training on some time series data. Naturally I want to sequentially mini-batch this data if at all possible. However, it seems that if the data size isn't a multiple of ...
3
votes
0answers
201 views

Hessian-Free instead of LSTM for Recurrent Net Machine Translation

Last year, Ilya Sutskever and collaborators came out with a paper about a recurrent LSTM net that learns sequence to sequence mappings for machine translation. It's somewhat surprising that the ...
3
votes
1answer
449 views

Stackable LSTM layer trained with arbitrary BPTT time steps

Anyone knows how to make a LSTM layer that is able to be trained with arbitrary BPTT time steps and easy to be stacked together? I am now implementing a basic version of LSTM layer. My scan function ...
2
votes
0answers
20 views

LSTM Generates Duplicates

I'm using an LSTM to generate molecules using the Simplified Molecular Input Line Entry System (SMILES) to represent molecules. As an example, Aspirin is represented as O=C(C)Oc1ccccc1C(=O)O I have ...
2
votes
0answers
72 views

Time Series Forecasting RNN: Difference Between Masking & Excluding Rows

Suppose you have missing values in a time series E.g. : t1 x1 y1 t2 ? ? t3 x3 y3 t4 ? ? t5 x5 y5 You are trying to forecast this time series using a recurrent ...
2
votes
1answer
156 views

How to paraphrase and augment training data for a question answering ML model?

I have only 50 question, answer pairs in my training data, where each question represent a unique intent. However, the training data is too small to build any meaningful ML model. What are the ...
2
votes
0answers
229 views

How to choose suitable Autoencoder (LSTM) architecture?

I am new to Autoencoders and I am a bit confused on which model to try for my situation and what is the difference between all the different models I have seen in tutorials. So, I have a set of time-...
2
votes
0answers
778 views

RNN LSTM overfitting

I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
2
votes
0answers
286 views

Awful performance of LSTM on noisy time series after stationarisation

Note. The post is quite long because I added some thought process for the sake of seeing the big picture. So grab a coffee and indulge yourself. For tldr the actual question on the bottom. I put my ...
2
votes
0answers
314 views

Vanishing Gradient on LSTM and GRU

LSTM and GRU are models that were proposed in order to solve the vanishing gradient issue. However, I have noticed that with long sequences these models also suffer from it, which makes sense. I am ...
2
votes
1answer
46 views

Dealing with the order of features (sequences)?

Assume we have following sequence database that is subsequently converted with one-hot encoding: 1 2 3 4 0 A B C D 1 B A D NA 2 A D C NA One-hot encoded: <...
2
votes
0answers
156 views

Training a bidirectional LSTM is unstable

I'm trying to solve timeseries classification problem. That's my model: ...
2
votes
0answers
99 views

LSTM - Learning a sinus function with linear part

I have recently build a simple LSTM-Network to predict a sinus function, which worked fine. Now I wanted to fit a sinus function containing a linear part with the same network but the results are ...
2
votes
0answers
140 views

Are there any rules of thumb for the number of hidden layer neurons in a RNN or LSTM for time series prediction?

Say that I have a univariate time series X(t) that I want to forecast using RNN/LSTM. I have 2 years of weekly sales data that is seasonal. How many hidden layers and neurons in each layer do I need ...
2
votes
0answers
39 views

Understanding Calculations in LSTMs

I’m trying understand LSTMs better: When we set the LSTM units in Keras or Tensorflow as: model.add(LSTM(256)) or ...
2
votes
0answers
134 views

LSTM good at hallucinating, useless at ground truth prediction?

I was interested in this project, so I cloned it and trained it on Moby Dick, for this challenge. The goal is to predict the next character given the past ground-truth characters. Overfitting is not ...
2
votes
1answer
434 views

LSTM network in the Asynchronous Advantage Actor-Critic (A3C) algorithm

I'm a little confused about the usage of LSTM network in the Asynchronous Advantage Actor-Critic (A3C) algorithm. The input for LSTM network is a sequence and network state, so my question is that ...
2
votes
0answers
2k views

LSTM for stock prices and trends prediction

I have an assignment to create a LSTM network predicting price and trend of cryptocurrencies based on stock market data from the past. The network I am using is a multilayered LSTM, where layers are ...
2
votes
0answers
1k views

How to use Keras pre-trained 'Embedding' layer?

guys! I've trained model in keras using Embedding on specific corpus of articles. I use this tutorial http://adventuresinmachinelearning.com/word2vec-keras-tutorial/ Now I want use it as layer in my ...
2
votes
1answer
138 views

understanding lstm results

I'm following an introduction to LSTM Neural Nets from this repo - https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Series-Prediction The author has a few ways to plot and test the ...
2
votes
1answer
955 views

LSTM extreme overfitting on learning rate reduction

I'm applying a single layer LSTM with hidden_size=16 towards a binary classification task. My training and validation loss are both reasonable until around epoch 400 when my learning rate gets halved, ...
2
votes
0answers
109 views

How to learn timeseries with LSTM, having different orders of magnitude in the output?

I am relatively new to neural networks and LSTM networks in particular, but I already worked with other ML algorithms. I am currently trying to reproduce a physical model (based on ordinary ...
2
votes
2answers
4k views

How to train a LSTM model for a next basket recommendation problem?

I try to use a LSTM model for a problem of next basket recommendation. I would like to apply the same approach as this article in Python using Keras : A Dynamic Recurrent Model for Next Basket ...
2
votes
0answers
673 views

LSTM for classification

I have a dataset which consists of $n_\text{samples}$ different measurements. Each measurement contains $n_\text{features}$ features. These features are for example ...
2
votes
0answers
670 views

Reinforcement Learning and Neural Networks with LSTM

I am working on a project training neural networks with an LSTM layer using Q-Learning. I haven't been able to achieve optimal results on my test bench problems. I believe my problem has to do with ...
2
votes
1answer
520 views

What's the difference between a single output RNN and a MLP whose input data contains all the features of given time steps?

When a RNN/LSTM contains output in each time step, I can understand that the output of current time steps is a function of its historical data. But when one deals with a RNN that only has an output ...
2
votes
0answers
228 views

Generating real-valued time series with RNN

I'm trying to adapt Andrej Karpathy's char-rnn model (which is also described in Alex Graves's Generating Sequences With Recurrent Neural Networks ) to real-valued time series. Unfortunately I'm ...
2
votes
0answers
392 views

Deriving Gradients for a Vanilla LSTM

I've been banging my head on this for far too long. The following code should be easy to understand; can someone assist me in discovering what I've done wrong? The code passes a numerical gradient ...
2
votes
0answers
131 views

When training an RNN, what are the important factors for deciding how many unrollings / unfoldings to use?

As far as I understand many RNN:s are trained with back propagation over a sequence of $k$ datapoints. The RNN is "unrolled" for each datapoint, i.e. its output is fed into itself together with the ...
2
votes
0answers
534 views

Encoding multiple states and events as time series

I have data on a collection of entities, along with data about things that have happened to them over time. I am trying to encode this into a categorical time series for use in an LSTM neural network ...
2
votes
0answers
167 views

Measure similarity between $2$ variable length sequences using LSTM

I would like to build a model that measures the similarity between $2$ sequences that have different lengths. Input: $[a_1, a_2, \dots, a_{n-1}, a_n]$ and $[b_1, b_2, \dots, b_{m-1}, b_m]$ where $a_i$...
2
votes
0answers
731 views

What is the output of word embedding in LSTM tutorial?

I'm new to theano and found LSTM tutorial's embedding word so confusing as lack of print function. in build_model function we have: ...
2
votes
0answers
83 views

A conceptual question about LSTM-RNN

I am working on series prediction by LSTM-RNN. In the training stage, I use a random series (white noise ) as input to go through a system and get the output. LSTM is implemented to learn the ...
2
votes
0answers
205 views

Training a binary LSTM classifier with few true positives

I'm trying to solve a multilabel classification problem with n-binary LSTM classifiers. I have 17 classes in total, where multiple classes may be true for each example (e.g. news articles with ...
2
votes
0answers
295 views

Difficulty learning parameters in RNN?

I'm implementing an LSTM using the RNN package in Torch. I've been able to get very simple models to converge (like learning the relation f(x) = x), but haven't been able to get basic things like ...
1
vote
0answers
34 views

Why do I get worse result after normalizing my input data of LSTM?

The first pic shows my data before normalization: And the second pic show my data after normalization using sklearn.preprocessing.normalize : I get better result ...