Questions tagged [lstm]

A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.

281 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
3k views

Which loss function to use when training LSTM for time series?

I'm experimenting with LSTM for time series prediction. The example I'm starting with uses mean squared error for training the network. I know that other time series forecasting tools use more "...
5
votes
0answers
40 views

Feature engineering for sheet music

I have a large dataset of digitized music scores that I'd like to use as input to a network. Initially, I'm looking to train networks to identify key signatures, tempo, dynamics, etc. from the raw ...
5
votes
0answers
207 views

How are the internal representations in ELMo averaged?

I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-...
5
votes
0answers
517 views

traditional state-space models and LSTMs

I am trying to understand the nature of LSTMs in relation to intuitions from traditional state-space models (e.g., Kalman filtering). The code below aims to simulate a simple univariate linear state-...
4
votes
0answers
2k views

Rolling window time series training and validation in Keras

I have a conceptual question regarding the use of the rolling window approach for training and validating a recurrent neural network (LSTM or GRU) on time series data. I have daily time series data ...
4
votes
0answers
964 views

Why is the derivative of the LSTM cell state w.r.t. to the previous cell state equal to the forget gate?

I keep seeing this online, on Quora and Machine Learning subreddits but I don't get it. Here's some basic math to show otherwise: We use this equation for the cell state: $c_t = f_t \odot c_t\__1 + ...
3
votes
0answers
23 views

What assumptions about time series data are neccessary to use a stateless LSTM?

Basically I am trying to model a time series using an LSTM layer, and I was wondering whether I should be using a stateful or stateless LSTM layer. More specifically I was wondering what assumptions ...
3
votes
1answer
42 views

Which machine learning model could be used for the following?

I am an experienced programmer but very new to machine learning. I have a data set that consists of about 50,000 sets of 2,000 ordered values. All of the values are floats normalised to between 0 and ...
3
votes
1answer
432 views

Adding noise to time series data to increase training data

I am dealing with a weekly time series forecasting problem and I am currently investigating the use of an LSTM to make a multi-step forecast for a univariate time series. I actually have a ...
3
votes
0answers
548 views

Awful performance of LSTM on noisy time series after stationarisation

Note. The post is quite long because I added some thought process for the sake of seeing the big picture. So grab a coffee and indulge yourself. For tldr the actual question on the bottom. I put my ...
3
votes
0answers
2k views

LSTM for stock prices and trends prediction

I have an assignment to create a LSTM network predicting price and trend of cryptocurrencies based on stock market data from the past. The network I am using is a multilayered LSTM, where layers are ...
3
votes
0answers
192 views

Is my weight matrix *learning* from all the steps in my LSTM?

I'm attempting to build an LSTM in Tensorflow to take in a series of amino acids (represented as Bitfields) and output a series of Torsion angles (4 numbers ranging from -1 to 1) for each amino acid ...
3
votes
0answers
839 views

When, if at all, to reset the state of an LSTM when training and when testing?

I am building an LSTM that takes in time-series financial data. My dataset is made up of IDs (each ID is a certain stock), and timestamps. For each ID at each timestamp, there are a number of features ...
3
votes
2answers
5k views

How to train a LSTM model for a next basket recommendation problem?

I try to use a LSTM model for a problem of next basket recommendation. I would like to apply the same approach as this article in Python using Keras : A Dynamic Recurrent Model for Next Basket ...
3
votes
0answers
403 views

Truncated Back Propagation of LSTM with K2 > K1

Every K1 time steps we will run truncated back prop through time (BPTT) for k2 time steps. For k2 <= k1 I don't find any problems. But let's look at the case where k2 > k1, and particularly where ...
3
votes
1answer
559 views

Mini-batching dependent data

I have a neural network I am training on some time series data. Naturally I want to sequentially mini-batch this data if at all possible. However, it seems that if the data size isn't a multiple of ...
3
votes
0answers
414 views

Deriving Gradients for a Vanilla LSTM

I've been banging my head on this for far too long. The following code should be easy to understand; can someone assist me in discovering what I've done wrong? The code passes a numerical gradient ...
3
votes
0answers
202 views

Hessian-Free instead of LSTM for Recurrent Net Machine Translation

Last year, Ilya Sutskever and collaborators came out with a paper about a recurrent LSTM net that learns sequence to sequence mappings for machine translation. It's somewhat surprising that the ...
3
votes
1answer
468 views

Stackable LSTM layer trained with arbitrary BPTT time steps

Anyone knows how to make a LSTM layer that is able to be trained with arbitrary BPTT time steps and easy to be stacked together? I am now implementing a basic version of LSTM layer. My scan function ...
2
votes
0answers
22 views

Application of Wavelet Transform and Differencing on Time Series Data (to denoise and remove seasonal adjustment and other trends)

I am working on an LSTM model to predict time series data (stock prices) and I would like an opinion whether to denoise my data or not before feeding it into the model. According to INVESTOPEDIA, ...
2
votes
1answer
22 views

Is it possible to overfitting within single epoch

Let me put my question first. For a time-series prediciton, is it possible to overfit even within the first epoch, when training data and validation data should all "new" to model? Features and ...
2
votes
0answers
23 views

Predict Future Sales

Which modelling strategy (time frame, features, modelling technique) would you recommend to forecast 3-month sales for total customer base? At my company, we often analyse the effect of e.g. ...
2
votes
0answers
43 views

LSTM Generates Duplicates

I'm using an LSTM to generate molecules using the Simplified Molecular Input Line Entry System (SMILES) to represent molecules. As an example, Aspirin is represented as O=C(C)Oc1ccccc1C(=O)O I have ...
2
votes
0answers
102 views

Do we get the best performance with “batch_size = 1”(especially for LSTM)?

In my experience, choosing batch_size = 1 gives the best result and choosing the batch_size = whole data number gives the worst. ...
2
votes
0answers
104 views

Time Series Forecasting RNN: Difference Between Masking & Excluding Rows

Suppose you have missing values in a time series E.g. : t1 x1 y1 t2 ? ? t3 x3 y3 t4 ? ? t5 x5 y5 You are trying to forecast this time series using a recurrent ...
2
votes
1answer
501 views

How to paraphrase and augment training data for a question answering ML model?

I have only 50 question, answer pairs in my training data, where each question represent a unique intent. However, the training data is too small to build any meaningful ML model. What are the ...
2
votes
0answers
279 views

How to choose suitable Autoencoder (LSTM) architecture?

I am new to Autoencoders and I am a bit confused on which model to try for my situation and what is the difference between all the different models I have seen in tutorials. So, I have a set of time-...
2
votes
0answers
1k views

RNN LSTM overfitting

I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
2
votes
0answers
531 views

Vanishing Gradient on LSTM and GRU

LSTM and GRU are models that were proposed in order to solve the vanishing gradient issue. However, I have noticed that with long sequences these models also suffer from it, which makes sense. I am ...
2
votes
1answer
56 views

Dealing with the order of features (sequences)?

Assume we have following sequence database that is subsequently converted with one-hot encoding: 1 2 3 4 0 A B C D 1 B A D NA 2 A D C NA One-hot encoded: <...
2
votes
0answers
156 views

Training Bidirectional Recurrent Networks on very long sequences

A common way to train Recurrent Neural Networks (RNNs) is back-propagation through time (BPTT), whereby the entire computational graph is 'unrolled' along each timestep and the derivatives can be ...
2
votes
0answers
186 views

Training a bidirectional LSTM is unstable

I'm trying to solve timeseries classification problem. That's my model: ...
2
votes
0answers
454 views

Mostly zero valued data and LSTM network

I'm training an LSTM neural network to predict multiple timeseries. However, most of my train timeseries have a lot of zero values. This makes the model good in predicting the future values of mostly ...
2
votes
0answers
115 views

LSTM - Learning a sinus function with linear part

I have recently build a simple LSTM-Network to predict a sinus function, which worked fine. Now I wanted to fit a sinus function containing a linear part with the same network but the results are ...
2
votes
0answers
187 views

Dimmensionality reduction for highly dimensional multivariate time series with few time steps

I am looking for a dimmensionality reduction technique that is ideally compatible with LSTM. The dataset I am working with is a multivariate time series with 10 time points and ~13000 features. The ...
2
votes
0answers
156 views

Are there any rules of thumb for the number of hidden layer neurons in a RNN or LSTM for time series prediction?

Say that I have a univariate time series X(t) that I want to forecast using RNN/LSTM. I have 2 years of weekly sales data that is seasonal. How many hidden layers and neurons in each layer do I need ...
2
votes
0answers
41 views

Understanding Calculations in LSTMs

I’m trying understand LSTMs better: When we set the LSTM units in Keras or Tensorflow as: model.add(LSTM(256)) or ...
2
votes
0answers
172 views

LSTM good at hallucinating, useless at ground truth prediction?

I was interested in this project, so I cloned it and trained it on Moby Dick, for this challenge. The goal is to predict the next character given the past ground-truth characters. Overfitting is not ...
2
votes
1answer
738 views

LSTM network in the Asynchronous Advantage Actor-Critic (A3C) algorithm

I'm a little confused about the usage of LSTM network in the Asynchronous Advantage Actor-Critic (A3C) algorithm. The input for LSTM network is a sequence and network state, so my question is that ...
2
votes
0answers
1k views

How to use Keras pre-trained 'Embedding' layer?

guys! I've trained model in keras using Embedding on specific corpus of articles. I use this tutorial http://adventuresinmachinelearning.com/word2vec-keras-tutorial/ Now I want use it as layer in my ...
2
votes
1answer
2k views

LSTM extreme overfitting on learning rate reduction

I'm applying a single layer LSTM with hidden_size=16 towards a binary classification task. My training and validation loss are both reasonable until around epoch 400 when my learning rate gets halved, ...
2
votes
1answer
177 views

Classify the main semantic relation of a sentence using keras

I tried to ask in SO, but they told me to ask here. I have a big dataset like this: ...
2
votes
0answers
130 views

How to learn timeseries with LSTM, having different orders of magnitude in the output?

I am relatively new to neural networks and LSTM networks in particular, but I already worked with other ML algorithms. I am currently trying to reproduce a physical model (based on ordinary ...
2
votes
0answers
1k views

Adding weather forecast to RNN LSTM Keras for time series prediction

[worked on it for the last month] Assumptions: predicted value (demand for heat in a district heating system)(*) depends on: -weather -hour of the day -day of the week -past pattern (of the ...
2
votes
0answers
695 views

LSTM for classification

I have a dataset which consists of $n_\text{samples}$ different measurements. Each measurement contains $n_\text{features}$ features. These features are for example ...
2
votes
0answers
690 views

Reinforcement Learning and Neural Networks with LSTM

I am working on a project training neural networks with an LSTM layer using Q-Learning. I haven't been able to achieve optimal results on my test bench problems. I believe my problem has to do with ...
2
votes
0answers
233 views

Generating real-valued time series with RNN

I'm trying to adapt Andrej Karpathy's char-rnn model (which is also described in Alex Graves's Generating Sequences With Recurrent Neural Networks ) to real-valued time series. Unfortunately I'm ...
2
votes
0answers
134 views

When training an RNN, what are the important factors for deciding how many unrollings / unfoldings to use?

As far as I understand many RNN:s are trained with back propagation over a sequence of $k$ datapoints. The RNN is "unrolled" for each datapoint, i.e. its output is fed into itself together with the ...
2
votes
0answers
612 views

Encoding multiple states and events as time series

I have data on a collection of entities, along with data about things that have happened to them over time. I am trying to encode this into a categorical time series for use in an LSTM neural network ...
2
votes
0answers
191 views

Measure similarity between $2$ variable length sequences using LSTM

I would like to build a model that measures the similarity between $2$ sequences that have different lengths. Input: $[a_1, a_2, \dots, a_{n-1}, a_n]$ and $[b_1, b_2, \dots, b_{m-1}, b_m]$ where $a_i$...

1
2 3 4 5 6