Questions tagged [recurrent-neural-network]
A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.
639
questions
1
vote
1
answer
20
views
Gradient Clipping of Vanilla RNNs vs LSTMs
I am doing an online course that states that the reason we use LSTMs and similar variations of vanilla RNNs is because of the vanishing/exploding gradients problems with vanilla RNNs.
However, an ...
- 11
0
votes
0
answers
4
views
NN prediction of fault duration time
I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges:
Model Performance: My neural network seems to be ...
- 1
0
votes
0
answers
21
views
Strength of LSTM for text prediction
I am trying to understand the working of LSTM by considering an application. Given below is the application of LSTM for spam prediction a well known dataset.
...
- 123
0
votes
0
answers
14
views
Does LSTM keep its state between subsequent samples in the training process?
I'm not sure if I understood well how LSTM works.
I know that the LSTM layers keep an internal state. But I'm not sure if this state is maintained between subsequent samples during the training ...
- 678
0
votes
0
answers
10
views
Test results with trained LSTM model
I have 10,000 data for training of LSTM model. 10,000 dataset is extracted from database for the years before 2020. 60% of data is used for training, 20% for validation and 20% for test. Test data is ...
- 483
0
votes
0
answers
19
views
Can LSTMs/RNNs' Final Hidden States Serve as Effective Sequence Embeddings for Document or Sentence Analysis?
Can the final hidden state of LSTMs/RNNs be utilized as a sequence embedding for documents or sentences?
These embeddings could then be applied to tasks like classification or clustering to leverage ...
- 13
1
vote
0
answers
24
views
Why the number of weights in simple RNN is more than the number of weights in GRU? [closed]
I have read that the number of (trainable) weights/parameters are more in simple RNN compared to GRUs even though GRU also has gates present in it, can anyone explain? Thanks
- 11
0
votes
0
answers
6
views
Using multiple RNNs for different feature time intervals
I am currently working with a dataset that consists of time series data of 50 different parameter.
As usual, these time series are quite irregular. However, there are parameter groups that have a ...
- 101
1
vote
0
answers
56
views
Time Series Forecasting with multiple input, single output that has multiple values for each time step
I'm trying to understand if possible to generate a time series forecasting program that is able to deal with multiple values for each time step. Examples like this tensor flow guide show time series ...
- 11
0
votes
0
answers
9
views
Distribution of residual errors of RNNs
I have developed four RNNs (LSTM, GRU, BiLSTM and BiGRU). All of these models give residual errors with non-normal distribution. Is this acceptable? I know that for linear regression problems, we need ...
0
votes
0
answers
64
views
ML Modeling Recommendation for Predicting Snake Encounters in Historical Journey Data
I have a dataset consisting of historical journey data where individuals travel from point A to point B. During their journeys, they may encounter varying numbers of animal sightings, including snakes....
- 11
0
votes
0
answers
13
views
ML (NN) model for a physical process (layer movement in ensilage)
The problem is, I have a several silages with several layers of some substance in them (e.g. coal). Each layer has its own physical/chemical properties (concentration of element, X). Concentration of ...
- 111
0
votes
0
answers
19
views
If not taking into account data characteristics, are longer sequences generally harder for an LSTM-model to classify?
I understand that LSTM architectures are purposed for sequential data where an understanding of context could positively contribute to a prediction.
But if leaving this factor out (or, i.e., assuming ...
0
votes
0
answers
6
views
How can I learn sequence-to-sequence imputation model (SSIM and dual SSIM)?
I have read papers that have used this model for the imputation method.
Link: https://ivivan.com/papers/IOTJ2019.pdf
I want to read more and be able to use these models for the imputation of my own ...
0
votes
0
answers
24
views
Meaning of Skewness and Kurtosis values of Residual Errors in Time Series Forecasting Problem using LSTM
I have developed different kinds of RNNs (such as LSTM,GRU etc.)to predict future values of thermocouple measurements. The residual errors look like they do not follow normal distribution, so I wanted ...
1
vote
1
answer
49
views
Expressive power of RNNs and deep RNNs
In class, my professor that a recurrent neural network is for problems where the state changes as a function of past state and new input, i.e. h_(t+1) = f(h_t, x_t), for any f. Then he said we would ...
- 111
0
votes
0
answers
26
views
Advantages of Using Recurrent Neural Network (RNN) Over Other Machine Learning Algorithms for Classification Tasks
As far as I know, one of the main advantages of using RNN is to learn sequences of texts from the (textual) data set. If I am correct, this is intuitive to understand when RNN is applied to a task ...
- 151
0
votes
0
answers
45
views
Can a machine learning algorithm learn to differentiate a function?
As the question says, can a machine learning algorithm learn to differentiate a function?(Eg. If we give such a network a function $f(x)$, it should output its derivative $f'(x)$.)
Clearly, a simple ...
- 101
0
votes
0
answers
35
views
Models do not predict peak values
I am training three models, ANN, LSTM and RNN to predict the PV energy production.
The data contains two features which are the irradiation which is normalised (highest value 1) and the historical ...
- 101
0
votes
0
answers
19
views
Negative RSquare Value but Good View on Chart for RNN Time Series
I'm fairly new on univariate time series forecasting and deep learning.
My task is to forecast energy consumption values.
May data looks like:
I'm usin simple RNN, because I have tested with linear ...
- 1
0
votes
0
answers
13
views
Shortening LSTM Sequences
I am training an LSTM where I have sales data from 20 different individuals over the past 10 years.
Now, I read this brilliant answer: How to train LSTM model on multiple time series data?
But, due to ...
- 235
0
votes
1
answer
85
views
RNN/LSTM networks on spectrograms underfitting massively - is the CNN encoder a prerequisite?
I am prototyping a pipeline on the FSDD dataset (audio/10-class classification); the audio data are loaded with librosa, 0-padded/trimmed to 0.5 sec (4000-dimensioned numpy vectors) each and converted ...
- 81
1
vote
0
answers
221
views
What loss function should one use when using an RNN to to predict a time series of binary values which are not necessarily independent?
I'm trying to use an RNN to predict a time series {$y_t$}, where $y_t \in {0,1} \forall 1 \leq t \leq T$ given an input time series {$x_t$}. The elements of the target sequence are not necessarily ...
- 1,665
0
votes
0
answers
25
views
How are RNNs trained for infinite horizon reinforcement learning
In the literature in case of POMDPs, the policy is conditioned on the state and the observation/belief, which should summarize the history of the states/actions taken so far.
To do so, usually a RNN ...
0
votes
0
answers
26
views
Vanishing Gradient Problem: What is the cause from a Data perspective? [duplicate]
My question:
I know there exists a lot of information about what causes the vanishing gradient from a computational standpoint. Ie due to way the RNN is trained by backpropagation [...]. Why do RNNs ...
- 101
2
votes
0
answers
377
views
Recurrent neural networks vs. State space models
I'm trying to understand the differences between RNNs and State Space Models (SSMs). I know that SSMs can take on different definitions depending on who you ask, but here I define it as in Learning ...
- 313
0
votes
0
answers
16
views
Memory Cells vs Attention: Learnable Parameters?
Are weights and other parameters inside memory cells such as LSTMs and GRUs learned during backprop?
If they do get optimized and learned, what is the difference between these cells and an attention ...
- 1,014
2
votes
2
answers
104
views
Given enough time, can a fully connected layer approximate a causal convolution layer?
In the paper WaveNet: A Generative Model for Raw Audio, the authors try to capture spatial data as follows:
They do this by limiting the scope of the hidden layers to particular sections of the input,...
- 630
0
votes
0
answers
54
views
Why do training and fixing a reservoir yield very similar results (in an echo state network)?
I am trying to better understand how echo state networks work. To see, how fixing the weights of the reservoir of an echo state network impacts the prediction quality of an echo state network, I have ...
- 111
0
votes
0
answers
30
views
Seq2Seq LSTM or RNN with independent hidden state
I'm curious about sequence-to-sequence mapping using recurrence networks like LSTM and RNN. As I've seen so far, in machine translation using RNN or LSTM, people usually use an Encoder-Decoder network ...
0
votes
0
answers
19
views
Prediction model for [parameter vector] to [time series]
Say I have a function $F$ that takes in a parameter vector $P$ (say, a 5-element vector), and produces a (numerical) time series $Y[t]$ of length $T$ (eg $T$=100, so $t=1,...,100$). The function could ...
- 117
0
votes
0
answers
13
views
How does LSTM perform inference on untrimmed video or audio?
I am new to the concept of RNN and LSTM. The concept of LSTM gave me an impression that it performs inference step by step. If the input data is a video, it first consumes Frame(t-1), and then ...
- 113
0
votes
0
answers
44
views
RNN's Output Layer: How does it learn from its prev iterations if each Activation Vector is processed in parallel?
First of all, are the activation vectors processed in parallel? If so:
That doesnt make sense since each previous activation vector feeds into the RNN as input. So if you're processing all activation ...
- 704
0
votes
0
answers
46
views
Recurrent dropout in keras: is the Semeniuta method equivalent to the Gal and Ghahramani method from the point of view of variational inference?
as the title. from this post https://stackoverflow.com/questions/44924690/keras-the-difference-between-lstm-dropout-and-lstm-recurrent-dropout I have recently learned that the implementation of ...
- 325
1
vote
1
answer
90
views
How to calculate dimension of weight matrix for a vanilla RNN
Sorry if this is kind of a dumb question but I'm taking the NLP course from Andrew Ng on Coursera and I don't understand how to arrive at the correct answer for a question pertaining to calculating ...
- 11
0
votes
0
answers
37
views
Using RNN in predicting time series with regular gaps
I have time series data spanning over 25 years, but covering only about a 3.5 week window at the same time each year and I'm only concerned with predictions in the middle of this window. Would an RNN ...
- 123
0
votes
0
answers
33
views
How do I prepare data for a multivariate LSTM model that includes multiple patients
I want to predict the blood glucose levels using time series data with multiple features such as time, glucose levels, carbohydrates, fat, and protein. I have a dataset with hundreds of patients but ...
- 1
0
votes
0
answers
26
views
LSTM Hyperparameter tuning, reasonable range
As the title describes, I am not clear what a reasonable range for hyperparameter tuning is. I know that there is not a definite answer when it comes to hyperparameter tuning. However, I was wondering ...
- 645
1
vote
0
answers
82
views
Why LSTM predicts very poorly with loss curves showing no indication of overfitting or underfitting?
I am training a LSTM model for load demand prediction with:
Training Data: 18288 samples with 9 features
Validation Data: 0.05% of Training Data (about 915 samples)
Data Scaling: MinMax Scaler(0,1)---&...
- 11
2
votes
1
answer
70
views
Time series as one of several inputs to ML Model
I have what seems to be a relatively simple question that I haven't been able to find a satisfactory answer to despite searching for quite a long time:
The classic way to analyze time series data ...
- 143
0
votes
0
answers
23
views
How to improve low magnitude and seasonality-missing predictions in LSTMs?
I am using DeepAR to predict count (small integers > 0) time series of Electric vehicle demand at a charging station.
The prediction output on the test set is as follows:
Blue is the actual and ...
- 244
1
vote
0
answers
145
views
RNN with overlapping timestamp sequences
Maybe a newbie question here, but I’ve not had much experience with sequential models and I’ve not been able to find an example or clear answers to this question online.
All tutorials and resources I ...
1
vote
0
answers
20
views
Hopfield Network Energy
Can neurons in Hopfield Network have non-binary values ( continuous values instead of -1 and +1)? If they can have non-binary values , is energy expression for hopfield NN stays the same? What is the ...
- 11
1
vote
1
answer
73
views
Why is the notion of a batch problematic for RNNs?
This paper says that the notion of a batch problematic for RNNs (page 9) (which is why you can't apply batch normalization for RNNs?). Why is it hard to talk about batches for RNNs?
Eg. the Pytorch ...
- 119
1
vote
0
answers
64
views
Are time-series models essentially difference equations with noise?
I've been studying some difference equations in my free time, and now I'm seeing them everywhere, especially in time-series models.
Here are some models that seem to be difference equations to me. I'm ...
- 310
0
votes
0
answers
79
views
Can I use the raw output of a model as input to the same model?
I would like to train a machine learning model where the training data represents points along a path/trajectory in an environment for a robot to follow. Once trained, the model is responsible for ...
0
votes
0
answers
19
views
Design Neural Network for special star area
-Consider networks whose input is y,x (coordinates of the plane) and has only one hidden layer. Is it possible that this network is 1 for the inputs in the area below and 0 in the rest? Is Can the ...
0
votes
1
answer
166
views
What deep learning methods should I explore for my panel data where one country is fixed?
The data looks like the following image and is available from 1980 to 2019:
As you can see, nation 1 doesn't differ in any of the panels; only country 2 does and the country 2s for each panel remain ...
- 1
2
votes
1
answer
46
views
Shape of tensorflow model input
I'm reading Masking and padding with Keras, in the beginning, an input example is:
...
- 2,335
0
votes
0
answers
27
views
Input size vs hidden state in RNNs
Im using PyTorch to implement RNNs on univariate time series data. This is the documentation for the RNN class: link
I think I'm understanding the math behind an RNN cell. But I have an specific ...
- 183