Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [lstm]

A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.

2
votes
1answer
485 views

What's the difference between a single output RNN and a MLP whose input data contains all the features of given time steps?

When a RNN/LSTM contains output in each time step, I can understand that the output of current time steps is a function of its historical data. But when one deals with a RNN that only has an output ...
0
votes
0answers
7 views

Transformer based decoding

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder ...
0
votes
0answers
22 views

LSTM Autoencoders - architecture

I am a bit confused about the structure of LSTM autoencoders, as far as I know, common way to construct vanilla autoencoders is bottleneck structure, for instance, start with 40 nodes, encode it to 30 ...
0
votes
0answers
10 views

What to use as a “stop” signal in a hierarchical LSTM model with continuous variable-length outputs

I am implementing a hierarchical sequence-to-sequence deep neural network model using long short-term memory (LSTM), where the bottom level of the hierarchy generates discrete outputs (characters from ...
4
votes
3answers
90 views

Binary target prediction using LSTM with sparse events in time

I have a data of patients that have multiple events happening in there medical history, I'd like to predict a target of having a specific targeted-event in the next 30 days. The data is timestamped ...
0
votes
0answers
17 views

Recurrent Neural Network - Time Series prediction: How to decide on the sequence length and number of output neurons

Context I am currently working with Gated Recurrent Units for time series prediction. Anyhow, my question should apply to all kinds of such recurrent networks. In my data, I have timestamps of 15 ...
4
votes
1answer
4k views

LSTM cell output activation for series

This question is asked from the perspective of finding out if there's a more efficient way for an LSTM to act more as a regression entity rather than just assigning only probabilities to the next ...
0
votes
1answer
201 views

Normalization for MFCC?

I'm planning on using MFCCs extracted from audio signals to make a speaker recognizer. I noticed that the first MFCC term tends to be very large, compared to the others. That's why I think that ...
4
votes
1answer
502 views

What is the most efficient method to handle long time sequences (LSTM)?

I am using LSTM and I have several long time sequences of varying length. Most of them are about 6,000-7,000 timesteps on average, but several are around 40,000 long. I am not sure which of this would ...
0
votes
0answers
6 views

LSTM/RNN history-based prediction by using Keras backend?

For my experiment, I have a formatted csv file with 1440 columns like following: ...
1
vote
1answer
30 views

Predict song genre using LSTM

I have a dataset of songs based on genres. For example, a song may hold {5, 2, 3} as scores set for Sentimental, Rock and Jazz. In total there are 800 songs sequentially arranged. I want to predict ...
19
votes
3answers
19k views

Understanding input_shape parameter in LSTM with Keras

I'm trying to use the example described in the Keras documentation named "Stacked LSTM for sequence classification" (see code below) and can't figure out the ...
0
votes
1answer
184 views

About RNN with variable length output vectors

I have several thousands samples with equal number of features (5000, they are time dependent) and I would like to predict of vectors with variable length. 1) I'm beginner in RNN, and I'd like to ...
0
votes
0answers
13 views

How do I implement masking in TensorFlow eager?

I am training a stateful RNN on variable length sequences (optional: see my previous question for more details). I padded the sequences to a fixed length with the value -1. The when batches are ...
0
votes
1answer
24 views

How to do location forecasting on Chicago Crime Dataset?

I am using the dataset https://www.kaggle.com/currie32/crimes-in-chicago and given primary type of the crime I want to forecast the next location of crime. What approach should I follow ?
7
votes
2answers
6k views

How to make LSTM predict multiple time steps ahead?

I am trying to use a LSTM for time series prediction. The data streams in once per minute, but I would like to predict an hour ahead. There are two ways I can think of for going about this: Squash ...
5
votes
1answer
380 views

Adding Features To Time Series Model LSTM

I have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to ...
3
votes
1answer
445 views

Stackable LSTM layer trained with arbitrary BPTT time steps

Anyone knows how to make a LSTM layer that is able to be trained with arbitrary BPTT time steps and easy to be stacked together? I am now implementing a basic version of LSTM layer. My scan function ...
0
votes
1answer
27 views

Given the universal approximation theorem, why are LSTM better than feed forward neural networks at certain tasks?

Per the universal approximation theorem, feed forward neural networks can approximate any function up to an arbitrary level of precision on the domain that they are trained on, given a sufficient ...
0
votes
0answers
26 views

Sequence Classification Machine Learning Methods

I know that this question might have been asked before but I did not find any response so far. My problem is that I am doing Multi-class sequence classification with Stacked LSTM (3 layers) using ...
0
votes
1answer
768 views

Many-to-many or many-to-one LSTM when predicting a value derived from a sequence of features

Let's say I have a time series data set consisting of features that may correlate to whether or not the price of a stock will go up or down. Say these data points are at 5 minute intervals. I build an ...
0
votes
1answer
595 views

Use of shuffled dataset for training and validating lstm recurrent neural network models

I am trying to build a recurrent neural net model using lstm trying to predict future outputs from a financial time series. Outputs are classified in macro classes according to the magnitude of ...
1
vote
0answers
13 views

LSTM repeatedly giving the same words

I implemented a 2-layer LSTM based on the architecture described in https://panderson.me/up-down-attention/ I trained it with Cross Entropy, and for every step I fed in the correct previous word (...
0
votes
0answers
7 views

Training and validation loss: consistency and interpretation

I have following training & validation loss for my LSTM network. I was wondering what I could deduce from this data. The validation loss seems to start where the training loss ends, is this a ...
1
vote
1answer
487 views

Recurrent Neural Networks: How to find the optimal parameters?

I am trying to fit a recurrent neural network for a binary classification problem using the 'keras' library (https://keras.io/layers/recurrent/). Now these networks have many parameters to tune. LSTM ...
0
votes
0answers
19 views

Is it okay to calculate all the gradients for an LSTM at once?

I'm trying to use the AC method reported in "Asynchronous Methods for Deep Reinforcement Learning" for a project. The relevant algorithm is shown in pseudocode at the bottom, Algorithm S3. I'm using ...
2
votes
1answer
127 views

understanding lstm results

I'm following an introduction to LSTM Neural Nets from this repo - https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Series-Prediction The author has a few ways to plot and test the ...
4
votes
0answers
274 views

traditional state-space models and LSTMs

I am trying to understand the nature of LSTMs in relation to intuitions from traditional state-space models (e.g., Kalman filtering). The code below aims to simulate a simple univariate linear state-...
2
votes
1answer
842 views

LSTM extreme overfitting on learning rate reduction

I'm applying a single layer LSTM with hidden_size=16 towards a binary classification task. My training and validation loss are both reasonable until around epoch 400 when my learning rate gets halved, ...
0
votes
1answer
17 views

What is the output of an LSTM

I have two questions regarding LSTMs: 1) Are LSTMs outputs' have shape/size exactly similar to the input? 2) Can we use LSTMs' intermediate outputs to deduce some sort of predictions? Context: ...
3
votes
1answer
202 views

How to minimize sharpe ratio with LSTM recurrent neural network?

I've read some articles about trading using recurrent reinforcement learning such as this one. The point where I do not fully understand is how to construct the cost/loss function. In the article, ...
4
votes
0answers
29 views

Feature engineering for sheet music

I have a large dataset of digitized music scores that I'd like to use as input to a network. Initially, I'm looking to train networks to identify key signatures, tempo, dynamics, etc. from the raw ...
0
votes
0answers
7 views

Using RNNs to segment long streams

I'm using an LSTM to segment a long streams of linear data into contiguous parts. My specific application is extracting key signature changes from a stream of notes. Specifically, I want to (for each ...
1
vote
1answer
28 views

Examples of “one to many” for RNN/LSTM

Are there any examples dealing with "one to many" kind of LSTM? Basically I am trying to build a model which takes an input vector $a$ and gives an output of $[b_1; b_2 ;b_3; b_4, \ldots; b_n]$ where ...
0
votes
0answers
8 views
3
votes
1answer
456 views

Mini-batching dependent data

I have a neural network I am training on some time series data. Naturally I want to sequentially mini-batch this data if at all possible. However, it seems that if the data size isn't a multiple of ...
0
votes
0answers
21 views

Understanding epoch, batch size, accuracy ,performance gain in lstm forecasting model

I am new to machine learning and lstm. I am referring this link LSTM for multistep forecasting for Encoder-Decoder LSTM Model With Multivariate Input section. Here ...
2
votes
2answers
92 views

How does an LSTM process sequences longer than its memory?

* Note: The premise of my question was incorrect in the first place. My question assumes that an LSTM maintains a separate set of weights for each time step in the memory it is given as a design ...
1
vote
1answer
844 views

LSTM Fitting Random Walk

I've got a question about an LSTM neural net fitting a random walk. I've made the LSTM [network shape: 1, 50, 100, 200, 50, 1] and out of interest made a completely random walk (by using a normal ...
0
votes
1answer
137 views

Parameters Grid Search for Keras LSTM on Time Series

How do you do grid search for Keras LSTM on time series? I have seen various possible solutions, some recommend to do it manually with for loops, some say to use scikit-learn GridSearchCV. Feedback ...
0
votes
0answers
16 views

RNN to predict completion of fixed-length time series

I need advice to model a certain kind of time series prediction for which I didn't find any existing solution. I have a large set of independent time series of fixed length (let's say 100 steps). All ...
0
votes
1answer
2k views

Keras - text classification, overfitting, and how to improve my model?

i am developing a text classification neural network based on this two articles - https://github.com/jiegzhan/multi-class-text-classification-cnn-rnn https://machinelearningmastery.com/sequence-...
0
votes
0answers
22 views

Tree-Paths as sequence input into a neural network

I'm currently trying to understand this paper but I struggle with the input into the NN. What I don't understand is what the input vectors should look like for the network described in b) in the image ...
0
votes
0answers
30 views

LSTM - When to use sliding window in time series classification?

Say I have a tensor of data with shape (30, 16000, 38) - where each tuple element corresponds to ...
0
votes
1answer
32 views

Attention for short sequence length. Is it reasonable?

Will the attention mechanism be useful for the short sequence length? Let's say your training corpus has each query of MAX length 10. and most queries are of word length 3-4 words. How reasonable is ...
0
votes
1answer
78 views

What are paralell training and attention mechanism?

I read a quite interesting paper here: http://hanj.cs.illinois.edu/pdf/kdd18_cyang.pdf Accordingly, the basic idea is to combine clustering and churn prediction so that it can imply some insight from ...
0
votes
0answers
7 views

when not using teacher forcing does backpropagation flow through input layers to output layers?

When training a Recurrent Neural Network that feeds outputs in as the next input(no teacher forcing) do gradients flow through input layers to output layers and also through the hidden states?
0
votes
0answers
29 views

Attention mechanism in LSTM model to predict next action

I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict ...
2
votes
1answer
1k views

Sequence lengths in LSTM / BiLSTMs and overfitting

I'm currently working with LSTMs and BiLSTMs, using Keras as library (TF backend). Following Tutorials and reading some papers, I found out that the sequences used are mostly quite short. What I do ...
0
votes
1answer
44 views

Binary Classification of Numeric Sequences with Keras and LSTMs [duplicate]

I'm attempting to use a sequence of numbers (of fixed length) in order to predict a binary output (either 1 or 0) using Keras and a recurrent neural network. Each training example/sequence has 10 ...