A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.
0
votes
0answers
9 views
(Why) Is the ouput's dimension in a LSTM bound to the number of recurrent units, and how are recurrent outputs passed?
Background: Looking for specific/interesting information on the equations within LSTM-Networks, I found the paper LSTM: A Search Space Odyssey. It is frequently mentioned in other articles. To gain a ...
0
votes
0answers
10 views
Mean Test Set Performance of LSTM & Evaluation
In the paper "Greff et al, 2017 - LSTM A Search Space Odyssey" they evaluate different variants of LSTM architectures against different tasks/datasets.
Could you help me to understand the evaluation?
...
0
votes
1answer
15 views
Testing an LSTM making predictions 1 timestep into the future
Say I have a time series data set of 100 sequential timesteps, and I want to train and test an LSTM on the data set, but only on forecasting a single timestep into the future.
I want more than one ...
0
votes
0answers
7 views
Support Vector Machines VS LSTMs: How well it is justifiable to use LSTM for its Generalization properties?
The question is pretty straightforward, How well one can justify using LSTMs(Neural Networks) for text classification task in terms of "Generalization" compared to classic support vector machines(SVM) ...
1
vote
2answers
47 views
How to define a time series classification problem?
I have 3 sets of time series data generated from sensors, I believe they have some correlation themselves. Certain "modes" of the system can be defined from the patterns from these signals. The signal ...
0
votes
0answers
21 views
LSTM model using Keras, two layers, each layer with different size, make sense? [closed]
a friend of mine asked me for help, and asked if the scenario is legal:
First Lstm hidden layer size is: 256, and the second hidden layer is: 128.
This is the code he sent me (Keras):
...
0
votes
1answer
26 views
Deep learning with a lot of training data
I am building a bidirectional LSTM to do a sequential text-tagging task (particularly, automatic punctuation). Usually, the training is done in iterations, where in each iteration, the entire training ...
1
vote
0answers
11 views
Interpreting accuracy graph for a LSTM model | Keyword Prediction
I have created a LSTM model for keyword prediction. I am using RMSE optimizer for training. I observe that the train and test accuracy decreases at first and then fluctuates without much difference in ...
0
votes
0answers
21 views
Best practices to apply Layer normalization in recurrent networks
I'm trying to add layer normalization (in the encoder-level) to the Listen-attend-and-spell model for speech recognition tasks.
To do so, I have done many experiments (all of them failed) to make my ...
0
votes
0answers
14 views
NaN output from Bidirectional LSTM in Keras
I am creating a bi-directional LSTM using tf.keras APIs:
input_layer = tf.reshape(emb_seqs, [const.TRN_BATCH_SIZE, -1, const.VECTOR_SIZE])
lstm_layer = tf.keras.layers.LSTM(units=LSTM_UNITS, ...
1
vote
0answers
42 views
How does LSTM learn [duplicate]
I am confused on how LSTM learn from word embedding. I know that LSTM accepts 3D input (sample, timesteps, features). So, when we use embedding layer (word2vec) and we have 300-d vector representation,...
0
votes
0answers
23 views
How to understand what is going on based on the loss curves?
I have been implementing a many-to-one LSTM and have been searching for the best hyperparameters. However, sometimes I am a little confused on the reason the loss acts as it does. Here is a screenshot ...
1
vote
1answer
14 views
Getting ValueError while implementing LSTM in keras
I am getting this error while implementing LSTM in Keras:
""""Error when checking input: expected lstm_16_input to have 3 dimensions, but got array with shape (156060, 1)""""
I have 156060 text ...
0
votes
1answer
6 views
Clarification about RNN/LSTM Sequence Models with Word Vector inputs
Say that we are trying to train a language model with an RNN/LSTM i.e. the inputs are words in a sentence and the outputs are the same words shifted by one such that for each input word the output is ...
0
votes
0answers
16 views
How to overfit an LSTM for acoustic features?
Aim
I would like to train an LSTM to learn the mapping between the EMA signals and audio signals in the MOCHA-TIMIT dataset. I've looked at publications using similar approaches to see if I can ...
0
votes
0answers
12 views
Different ways to deepen LSTMs
Quick introduction of the notation I'm working with:
With regular RNNs we have
$$h^{\left\langle t\right\rangle }=g_{h}\left(W_{h}\left[h^{\left\langle t-1\right\rangle },x^{\left\langle t\right\...
0
votes
0answers
92 views
Loss decreasing but highly oscillating over all training examples
I'm training a sequence model. I have 40 sequences of length 70,000. I have divided these into minibatches of size $[40, 200]$. Since I want to utilise the stateful nature of the GRU I'm training, I ...
4
votes
1answer
81 views
State-of-the-art algorithms for the training of neural networks with GRU or LSTM units
I recently read a lot about neural networks using GRU or LSTM units. There are many easy to use frameworks like tensorflow that do not even require high knowledge about programming. Unfortunately, I ...
1
vote
0answers
10 views
Getting OSerror while loading my model in keras [closed]
I created a model in keras and then save it to h5 file then I created a package for that model and used load_model function of keras to load that saved model of mine but I am getting this error:
""...
1
vote
0answers
22 views
LSTM Training loss decreases and increases
I am new to LSTM and deep learning. I have 3000 reviews which I am trying to train on gensim pretrained model via word embedding. I have the following model where ...
0
votes
0answers
16 views
Using an LSTM unrolled into multiple timesteps for training, and just a single timestep for inference
I have built an LSTM in TensorFlow that is unrolled into n timesteps and trains successfully on sequences of lengths n.
I would now like to use my trained model to make m daily predictions one by one....
1
vote
1answer
36 views
Why do we need second tanh() in LSTM cell
My question is why do we apply tanh() function to C second time when it has already been applied to it during the update procedure or, in case if we didn't update it, in the previous LSTM cell. I mean ...
0
votes
0answers
5 views
Find association rules between events and a failure
I am dealing with machine that provide logs of its current activity. These logs have over 4000 different messages (Info,Warning,Error,...).
I would like to predict a failure of that machine using ...
0
votes
0answers
21 views
Word embedding as input or raw text?
I'm trying to implement a neural network for text recognition and I'm a little bit confused about text inputs.
The goal of the network is to classify a comment, toy example:
...
1
vote
1answer
17 views
User behavior prediction using LSTMs
Let a user U with 3 possible states: A, B and C. From A you can go everywhere (including A), from B or C you can only go to A. LSTM are a good to model Markov-problems with an extra notion of long ...
0
votes
0answers
14 views
Multiple output NN vs multiple single output NN [duplicate]
I'm trying to build a 5 input-5 output model using LSTM, where all the outputs are the same features as the inputs, predicted in the future.
My question is: is it better to build 5 models, each with ...
1
vote
0answers
9 views
Best method for predicting a binary DV from multiple IVs with time series data in R or Python?
Let's say I have a dataset of 100.000 cases that contains 5 variables recorded over a period of time in long format, like this:
...
1
vote
1answer
49 views
Feedforward Nets, RNNs, and LSTMs Theory
It is my understanding that feed-forward neural nets learn by backpropegating the error from comparing an outcome to the ground truth. How is this process not 'recurrent', as in recurrent nets, data ...
0
votes
1answer
25 views
What neural network to use to classify time series data?
I am a self-learning machine learning, so this question may be a bit trivial. I've been reading "Python Machine Learning" by Sebastian Raschka and Vahid Mirjalili; however, I am still uncertain what ...
0
votes
1answer
44 views
2
votes
1answer
18 views
What is the reason that reduce training time over epoch for LSTM?
I am training and recurrent neural network and observed less time is needed over time.
What could be the reason? I would think calculating the gradient, and update the parameters in the network would ...
0
votes
0answers
13 views
Train on Sequential and Make Predictions on Non-Sequential Data using LSTM architecture in Keras
I am working with multivariate time series data with multiple examples to train LSTM on, and Y is either 0 or 1 binary classification. Currently, I am using pad sequence layer in combination with a ...
6
votes
2answers
141 views
Difference between a single unit LSTM and 3-unit LSTM neural network
The LSTM in the following Keras code
input_t = Input((4, 1))
output_t = LSTM(1)(input_t)
model = Model(inputs=input_t, outputs=output_t)
print(model.summary())
...
0
votes
0answers
21 views
Sequential Long-Text Classification with Recurrent and Convolutional Neural Networks
I am thinking to build a model for predicting events from news. Before I start this task I wanted to ask if someone have tried to build something like in the link(https://arxiv.org/pdf/1603.03827.pdf) ...
0
votes
0answers
73 views
How to choose suitable Autoencoder (LSTM) architecture?
I am new to Autoencoders and I am a bit confused on which model to try for my situation and what is the difference between all the different models I have seen in tutorials.
So, I have a set of time-...
2
votes
0answers
98 views
RNN LSTM overfitting
I'm trying to build a dynamic RNN network for 2-class classification, and I just can't get rid of the overfitting. I have 5500 samples of class A, and 8000 for class B (total 13500). From that I take ...
0
votes
0answers
6 views
How to pass MDLSTM activations to a feedforward layer?
I am currently trying to implement this paper by Alex Graves on Arabic handwriting recognition.
In the architecture presented in Figure 4 on page 11, it says that the image is passed to 4X2 MDLSTM, i....
0
votes
1answer
37 views
Generating sequences of musical chords
I'd like to create a model capable of emulating music that has been presented to it. The model ought to be specifically designed for that purpose, not just another generic, stacked LSTM. For the ...
1
vote
2answers
66 views
Vanishing Gradient for LSTM on State Classification Problem [closed]
I'm trying to do sequence classification on an LSTM. The problem setup is that we get data from a machine which is our input. We don't know what state the machine is in, so we perform classification ...
0
votes
0answers
19 views
LSTM frame time series to a supervised learning problem
I just begun to play around with LSTM. Therefore I read the guide from this site Multivariate Time Series Forecasting with LSTMs in Keras
The task is to predict the air pollution. I understand the ...
0
votes
0answers
10 views
How to sample a language model?
I've successfully trained a language model using LSTMs. But I have a confusion about sampling.
On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
0
votes
0answers
21 views
LSTM Training with High Score Variace
I am training a NN as conv -> conv -> dense -> lstm -> softmax, and got the scores/iteration as the figure shows. The dataset has been normalized with ...
1
vote
1answer
113 views
Is it possible that a stacked LSTM for text generation is performing worse than a one layer LSTM?
Does anyone know if the generated text from a stacked LSTM is performing worse than a one layer LSTM? Is a possible answer that the Model is overfitting?
In my case after the first few epochs the ...
2
votes
1answer
45 views
What is the purpose of unrolling an LSTM into multiple time steps if you can just use a stateful LSTM of 1 time step?
As far as I understand the follwoing two models are essentially identical:
Having a stateful LSTM with just a single time step and passing 10 time-series data points into it one by one, and using the ...
-1
votes
1answer
57 views
0
votes
0answers
31 views
LSTM Cell backpropagation through time & independence of gates
I am calculating the backpropagation across LSTM cell gates, and I notice that many online sources treat the gates as independent. For example, in this Aidan Gomez blog, he derives the backpropagation ...
1
vote
0answers
23 views
Deep learning unerfitting/overfitting
I started to perform deep learning for sentiment analysis on word embedding. I have plot the model loss and accuracy graph for each epochs to understand the performance better. I read the following ...
0
votes
0answers
11 views
Do LSTM models mimic using n-grams?
I understand that LSTM models mimic using n-grams because of the memory process it uses -- and that they are much more context oriented.
But is this only in cases for sequence to sequence predictions?...
2
votes
1answer
26 views
Extensions of LSTM for huge data
Consider dealing with a huge high frequency financial data forecasting, RNN/LSTM is a popular way to solve the task. But the problem is that say you have 1 million data points and you want to predict ...
2
votes
1answer
108 views
How to handle big vocabulary size with keras tokenizer?
I am actually working on a neural language model developed with keras.
I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
