Skip to main content

Questions tagged [recurrent-neural-network]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

Model training loss always converge to 1.35

I'm trying to create a multi-class classification model using RNNs. The input data has a sequence length of 90 and consists of 5 features, normalized to the [0,1] range. Here's the network ...
Mangi222's user avatar
2 votes
1 answer
41 views

Random Variables in the Neural Net diagrams

When I look at this discussion on linear regression: What is a random variable and what isn't in regression models It says: If we have the population regression function: $$Y_i = \beta_0 + \...
a12345's user avatar
  • 65
0 votes
0 answers
10 views

Gradient flow through sampled tokens when training RNNs (but without teacher forcing)

Suppose we want to train an autoregressive generative language model based on a recurrent neural network (RNN) architecture without teacher forcing: At each timestep, the RNN takes an input token $x_t$...
Ben JW's user avatar
  • 101
0 votes
0 answers
47 views

BPTT in multi-layer RNN

I am trying to understand the Backpropagation through time algorithm for multi-layer RNNs but I'm facing challenges in extending the process from single-layer to multi-layer architectures. I have ...
Alison's user avatar
  • 1
3 votes
1 answer
30 views

Do other outputs matter in RNN or only the final output counts?

I'm just exposed to RNN recently. But it's confusing that the network also generates seemingly redundant outputs other than the "hidden states" which are passed as previous memory for the ...
MathematicsBeginner's user avatar
1 vote
0 answers
8 views

Classification of intervals in time series data of multiple instances

I have a problem that I am trying to frame. I have signal data from ECG (a classic signal over time data). A close example here: https://github.com/jjongjjong/ECG_segmentation_1DUnet I am basically ...
PPenton's user avatar
  • 11
1 vote
0 answers
21 views

Test loss immediately goes up on LSTM

I'm trying to create an LSTM that predicts the sixth sports match for team A based on a sequence of 5 previous matches. My data is set up in a structure like this. Team A game 1 vs random team, team B ...
WILLYB's user avatar
  • 11
0 votes
0 answers
36 views

Forecasting RNN and LSTM without X_test

Dear StackExchange Community, My data is composed of only 1 time series variable (Stock prices of an asset) I have splitted it to train and test subsets. I have tarined an RNN and LSTM models with ...
Amirgiano's user avatar
0 votes
0 answers
17 views

Extremely Small Output Weight Values in Echo State Network

I have an echo state network that is producing an output weight matrix with extremely small output weights (on the order of 10^-200). Ideally, these weights should be within a more reasonable interval,...
Jonathan Frutschy's user avatar
1 vote
0 answers
42 views

Multilayer Perceptron vs. Recurrent Neural Network for Time Series Forecasting: Utilizing Multiple Lagged Values

I am currently analyzing daily sales data for a product sold across multiple stores using a Multilayer Perceptron (MLP) model. For simplicity, let's assume it consists of a single layer, structured as ...
Girigio's user avatar
  • 63
0 votes
0 answers
14 views

Is there any quick and easy "good" way to test if an LSTM is working (in the basic sense)?

I'm definitely a beginner and trying to get a handle on things so I apologise in advance for possible slowness of understanding :) Recently, I came across a problem where given some sequence of ...
R Mary's user avatar
  • 101
0 votes
0 answers
52 views

Simple RNN for predicting the next character [duplicate]

I implemented a simple RNN from scratch (using only the numpy library )for predicting the next characters, and I trained it on a simple text=“hello world”. It works ...
user avatar
0 votes
1 answer
32 views

Make Predictions with an RNN Using a Multi-dimensional Training Set

I have a 2D matrix TD of training data that is a collection of N non-linear signals that are functions of time (hence the ...
Jonathan Frutschy's user avatar
0 votes
0 answers
43 views

How to use LSTM, TFT, or other RNN with time series of different lengths

I have a dataset of financial transactions per day with the volume of the transaction and the price of the transaction. For each day I want to calculate the volume weighted average price at the end of ...
Mathematician....'s user avatar
0 votes
0 answers
19 views

How to prepare sequential word data for many-to-many LSTM?

I want to build a many-to-many LSTM and I need some guidance on how to prepare the data that I have for such a model. My data frame consists of subject_ID (dance couple), Time (in s, time step in a ...
Rūta Slivkaitė's user avatar
0 votes
0 answers
16 views

How to improve a NN for signal classification

I am using a RNN with two Bi-LSTM layers to classify signals. The signals are complex valued, so what I am feeding the network are the magnitude, phase and unwrapped phase. These parameters of the ...
DaDSPGuy's user avatar
0 votes
0 answers
17 views

Questions about data preprocessing for time series forecast

I have the following time series that I would like to forecast using a Recurrent Neural Network. You can download the csv file from my github here The task I want to solve is the following Given a ...
apt45's user avatar
  • 273
0 votes
0 answers
10 views

Model's training highly dependent on weights initialitation. How to deal with it?

I am training a simple RNN model in keras to predict a time series. The time series I am considering is just a sine function $$ f(t) = \sin \left(\frac{3}{10}t\...
apt45's user avatar
  • 273
0 votes
0 answers
34 views

Exploit temporal information in GCN for the Elliptic dataset

I am trying to reproduce the results obtained on the Elliptic dataset from [1] (the EvolveGCN) and [2] using PyTorch geometric and PyTorch Geometric temporal libraries. The problem I am facing is that ...
Luigi D'Amico's user avatar
0 votes
1 answer
139 views

RNN weight and state matrices

Implementations of RNN in NLP tasks, like those in https://dennybritz.com/posts/wildml/recurrent-neural-networks-tutorial-part-2/, are done using matrices, that are used to store the inputs, outputs, ...
Luis's user avatar
  • 1
2 votes
1 answer
55 views

Gradient Clipping of Vanilla RNNs vs LSTMs

I am doing an online course that states that the reason we use LSTMs and similar variations of vanilla RNNs is because of the vanishing/exploding gradients problems with vanilla RNNs. However, an ...
HelloWorld's user avatar
0 votes
0 answers
6 views

NN prediction of fault duration time

I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges: Model Performance: My neural network seems to be ...
Tools's user avatar
  • 1
0 votes
0 answers
23 views

Strength of LSTM for text prediction

I am trying to understand the working of LSTM by considering an application. Given below is the application of LSTM for spam prediction a well known dataset. ...
Regi Mathew's user avatar
0 votes
0 answers
20 views

Does LSTM keep its state between subsequent samples in the training process?

I'm not sure if I understood well how LSTM works. I know that the LSTM layers keep an internal state. But I'm not sure if this state is maintained between subsequent samples during the training ...
Zaratruta's user avatar
  • 1,018
0 votes
0 answers
18 views

Test results with trained LSTM model

I have 10,000 data for training of LSTM model. 10,000 dataset is extracted from database for the years before 2020. 60% of data is used for training, 20% for validation and 20% for test. Test data is ...
batuman's user avatar
  • 483
0 votes
0 answers
26 views

Can LSTMs/RNNs' Final Hidden States Serve as Effective Sequence Embeddings for Document or Sentence Analysis?

Can the final hidden state of LSTMs/RNNs be utilized as a sequence embedding for documents or sentences? These embeddings could then be applied to tasks like classification or clustering to leverage ...
palash behra's user avatar
1 vote
0 answers
41 views

Why the number of weights in simple RNN is more than the number of weights in GRU? [closed]

I have read that the number of (trainable) weights/parameters are more in simple RNN compared to GRUs even though GRU also has gates present in it, can anyone explain? Thanks
beginner's user avatar
0 votes
0 answers
21 views

Using multiple RNNs for different feature time intervals

I am currently working with a dataset that consists of time series data of 50 different parameter. As usual, these time series are quite irregular. However, there are parameter groups that have a ...
Ai4l2s's user avatar
  • 101
1 vote
0 answers
119 views

Time Series Forecasting with multiple input, single output that has multiple values for each time step

I'm trying to understand if possible to generate a time series forecasting program that is able to deal with multiple values for each time step. Examples like this tensor flow guide show time series ...
Orion's user avatar
  • 11
0 votes
0 answers
14 views

Distribution of residual errors of RNNs

I have developed four RNNs (LSTM, GRU, BiLSTM and BiGRU). All of these models give residual errors with non-normal distribution. Is this acceptable? I know that for linear regression problems, we need ...
MariaP's user avatar
  • 1
0 votes
0 answers
67 views

ML Modeling Recommendation for Predicting Snake Encounters in Historical Journey Data

I have a dataset consisting of historical journey data where individuals travel from point A to point B. During their journeys, they may encounter varying numbers of animal sightings, including snakes....
Sita's user avatar
  • 11
0 votes
0 answers
47 views

Meaning of Skewness and Kurtosis values of Residual Errors in Time Series Forecasting Problem using LSTM

I have developed different kinds of RNNs (such as LSTM,GRU etc.)to predict future values of thermocouple measurements. The residual errors look like they do not follow normal distribution, so I wanted ...
MariaP's user avatar
  • 1
1 vote
1 answer
103 views

Expressive power of RNNs and deep RNNs

In class, my professor that a recurrent neural network is for problems where the state changes as a function of past state and new input, i.e. h_(t+1) = f(h_t, x_t), for any f. Then he said we would ...
Joe C.'s user avatar
  • 111
0 votes
0 answers
73 views

Can a machine learning algorithm learn to differentiate a function?

As the question says, can a machine learning algorithm learn to differentiate a function?(Eg. If we give such a network a function $f(x)$, it should output its derivative $f'(x)$.) Clearly, a simple ...
Dedados's user avatar
  • 101
0 votes
0 answers
25 views

Negative RSquare Value but Good View on Chart for RNN Time Series

I'm fairly new on univariate time series forecasting and deep learning. My task is to forecast energy consumption values. May data looks like: I'm usin simple RNN, because I have tested with linear ...
Beyza's user avatar
  • 1
0 votes
1 answer
185 views

RNN/LSTM networks on spectrograms underfitting massively - is the CNN encoder a prerequisite?

I am prototyping a pipeline on the FSDD dataset (audio/10-class classification); the audio data are loaded with librosa, 0-padded/trimmed to 0.5 sec (4000-dimensioned numpy vectors) each and converted ...
Nikos H.'s user avatar
1 vote
0 answers
336 views

What loss function should one use when using an RNN to to predict a time series of binary values which are not necessarily independent?

I'm trying to use an RNN to predict a time series {$y_t$}, where $y_t \in {0,1} \forall 1 \leq t \leq T$ given an input time series {$x_t$}. The elements of the target sequence are not necessarily ...
stochasticmrfox's user avatar
5 votes
0 answers
2k views

Recurrent neural networks vs. State space models

I'm trying to understand the differences between RNNs and State Space Models (SSMs). I know that SSMs can take on different definitions depending on who you ask, but here I define it as in Learning ...
paul's user avatar
  • 405
3 votes
2 answers
160 views

Given enough time, can a fully connected layer approximate a causal convolution layer?

In the paper WaveNet: A Generative Model for Raw Audio, the authors try to capture spatial data as follows: They do this by limiting the scope of the hidden layers to particular sections of the input,...
AlanSTACK's user avatar
  • 640
0 votes
0 answers
46 views

RNN's Output Layer: How does it learn from its prev iterations if each Activation Vector is processed in parallel?

First of all, are the activation vectors processed in parallel? If so: That doesnt make sense since each previous activation vector feeds into the RNN as input. So if you're processing all activation ...
Katsu's user avatar
  • 991
1 vote
1 answer
262 views

How to calculate dimension of weight matrix for a vanilla RNN

Sorry if this is kind of a dumb question but I'm taking the NLP course from Andrew Ng on Coursera and I don't understand how to arrive at the correct answer for a question pertaining to calculating ...
Isaac's user avatar
  • 11
1 vote
0 answers
125 views

Why LSTM predicts very poorly with loss curves showing no indication of overfitting or underfitting?

I am training a LSTM model for load demand prediction with: Training Data: 18288 samples with 9 features Validation Data: 0.05% of Training Data (about 915 samples) Data Scaling: MinMax Scaler(0,1)---&...
Abdullah's user avatar
2 votes
1 answer
211 views

Time series as one of several inputs to ML Model

I have what seems to be a relatively simple question that I haven't been able to find a satisfactory answer to despite searching for quite a long time: The classic way to analyze time series data ...
jgholder's user avatar
  • 143
1 vote
0 answers
203 views

RNN with overlapping timestamp sequences

Maybe a newbie question here, but I’ve not had much experience with sequential models and I’ve not been able to find an example or clear answers to this question online. All tutorials and resources I ...
Jonathan Hill's user avatar
1 vote
0 answers
34 views

Hopfield Network Energy

Can neurons in Hopfield Network have non-binary values ( continuous values instead of -1 and +1)? If they can have non-binary values , is energy expression for hopfield NN stays the same? What is the ...
gray KK's user avatar
  • 11
3 votes
1 answer
245 views

Why is the notion of a batch problematic for RNNs?

This paper says that the notion of a batch problematic for RNNs (page 9) (which is why you can't apply batch normalization for RNNs?). Why is it hard to talk about batches for RNNs? Eg. the Pytorch ...
étale-cohomology's user avatar
1 vote
0 answers
88 views

Are time-series models essentially difference equations with noise?

I've been studying some difference equations in my free time, and now I'm seeing them everywhere, especially in time-series models. Here are some models that seem to be difference equations to me. I'm ...
ForceBru's user avatar
  • 310
0 votes
1 answer
227 views

What deep learning methods should I explore for my panel data where one country is fixed?

The data looks like the following image and is available from 1980 to 2019: As you can see, nation 1 doesn't differ in any of the panels; only country 2 does and the country 2s for each panel remain ...
learner's user avatar
2 votes
1 answer
48 views

Shape of tensorflow model input

I'm reading Masking and padding with Keras, in the beginning, an input example is: ...
CyberPlayerOne's user avatar
1 vote
0 answers
54 views

Calculation of Sensitivity of the Outputs to the Inputs over Time in RNN

Alex Graves, author of Supervised Sequence Labelling with Recurrent Neural Networks, described how sensitivity of an output to the inputs is calculated as follows: /* The delta terms refer to the ...
DeepGreenHand's user avatar

1
2 3 4 5
13