Questions tagged [recurrent-neural-network]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

How does attention add expressive power to encoder-decoders?

I am learning about the attention mechanism for the first time, and the context in which it has been introduced (watching Lecture 8 of Stanford's CS224N) is that of language translation using seq2seq ...
user avatar
0 votes
0 answers
10 views

States in RNN: what if they are measured

A simple RNN will take the output and feed it back to a state variable. The next output is then the function of input and state. Now imagine I want to use the "state" variable in RNN as an ...
user avatar
0 votes
0 answers
26 views

RNN loss function spikes and slow decrease

I'm training a rnn model for classification (many to one) : ...
user avatar
  • 1
0 votes
0 answers
28 views

Understanding types of LSTM and their use cases

I'm currently considering to use RNN/LSTM for a predictive modelling project that involves time-variant points. From looking at the following types of LSTM/RNN (in the picture below), I want to try ...
user avatar
0 votes
0 answers
19 views

Time series forecasting for revenue forecasting?

I am currently working on a project where I have to forecast the revenue for (the duration of) projects within the organisation. The organisation has several departments that occupy themselves with a ...
user avatar
  • 1
0 votes
0 answers
18 views

Similar results between feedforward neural networks, recurrent neural network and LSTM for time series data - Is this standard?

Tl;dr: I have trained feedforward neural networks, recurrent neural networks and LSTM networks to predict behaviour of weather temperature. The results are almost all the same (see below). Is this ...
user avatar
  • 111
0 votes
0 answers
4 views

Question about understanding Weights of Keras LSTM model

I am implementing Federated Learning (FL) using Keras LSTM. Starting with the simple example where multiple models are trained at different clients. Each client shares their model weights with the ...
user avatar
  • 382
0 votes
0 answers
26 views

Evaluation for LSTM model

I have created a model for text generation using LSTM. I am having chess sequences learned, reporting only the pieces moved during the moves. So when I move a pawn on my game there will be "p&...
user avatar
0 votes
0 answers
21 views

Mini batches and loss in recurrent neural networks (RNNs)

Suppose that we have a sequence $\left\{x^{(k)}\right\}_{k = 1}^{N}$ and that we wish to use a RNN to predict the next element of the sequence given the previous elements of the sequence (e.g., a ...
user avatar
  • 101
0 votes
0 answers
13 views

How to predict a mathematical progression with keras

I try the following model for a many-to-many recurrent network: ...
user avatar
0 votes
0 answers
7 views

Vector to sequence RNNs: do they take a random initial "prompt"?

I am going through the Deep Learning book by Ian Goodfellow (here) and came by the architecture for a vector to sequence RNN (Figure 10.9). I am not sure I understand how this architecture works and ...
user avatar
1 vote
0 answers
20 views

How to train LSTM model for single time series prediction but with multiple dataframes? [closed]

I have to predict whether the shot will be disruptive or non-disruptive. What I have is time-series data of both the types of shots and I want the model to predict using initial period data whether it ...
user avatar
0 votes
1 answer
31 views

If I use transformers nn, I don’t need to use CTC loss and language models?

In the creation of an ASR (Automatic Speech Recognition) using transformers neural networks, we don’t need to use a CTC loss and language models, do we? I’ve seen that RNNs do need to use language ...
user avatar
3 votes
0 answers
31 views

Stacking recurrent neural network layers horizontally vs vertically

In the top answer on this post: What are the advantages of stacking multiple LSTMs? the idea of stacking LSTMs vertically is distinguished from stacking them horizontally. I quite don't understand ...
user avatar
  • 31
0 votes
0 answers
13 views

Is the recursive form in a transformer efficient for long sequences?

I am reading the paper Transformers are RNNs:Fast Autoregressive Transformers with Linear Attention, where the authors propose the linear transformer, which seems to be very efficient in practice. Yet,...
user avatar
0 votes
0 answers
15 views

Trying to predict the next ignition off for a vehicle

Overview: I have a dataset that captures 6 months of data for 1000 cars. Each car is represented by it's unique identifier. The data captures the exact timestamp when the car was turned ON and turned ...
user avatar
  • 1
0 votes
0 answers
15 views

Evolving action and selection for automated agents

I have been reading this paper 1, and I came through this graph, which I had a hard time understanding. Basically, we have an environment with food and poison, and water, our agent has three sensors, ...
user avatar
1 vote
1 answer
78 views

Backpropagation through time: dimensions in the chain rule

Take the first step $t=1$ for simplicity, \begin{align} h_1 &= \text{tanh}(W_{hx}^T e_1 + W_{hh}^T h_0) \end{align} Usually it is written in convention: \begin{align} \frac{\partial L_1}{\...
user avatar
  • 308
0 votes
1 answer
29 views

Sequential recommendation: how to effective encoding output item?

Now I am learning about sequential recommendation - session based recommendation. I have understood that User-item interactions may be viewed as sequential action (first I clicked item A, then click ...
user avatar
  • 244
1 vote
1 answer
89 views

ANN vs RNN for time-series prediction?

The basic argument against ANN for time-series is that there is no conception of time for the inputs. However, it is obvious that one can feed multiple time-steps into an ANN by using various lag ...
user avatar
0 votes
2 answers
31 views

What degree of difference does validation and training loss need to have to be called overfit?

I've trained an LSTM network to predict time series data however i'm quite new to LSTMs and am unsure if the model has overfit. I know that an increasing validation loss relative to a decreasing ...
user avatar
2 votes
1 answer
64 views

Validity of basic train - test - split for a time series using a RNN

I am trying to determine if a simple train-test-split is valid for a time series if I use a Recurrent Neural Network (LSTM). Lets say I have samples (x) which consist of 2 days values (time steps) ...
user avatar
  • 23
0 votes
1 answer
39 views

What are the transition functions for RNNs

From what I understand, the hidden states of RNNs are equivalent to the deterministic probability distribution over hidden states in for example a Hidden Markov Model. Thus, just as probabilistic ...
user avatar
0 votes
0 answers
13 views

Calculating RNN loss (for a SINGLE example) as a sum of individual time step losses VS. an average of individual time step losses [duplicate]

In Andrew Ng's course, I see RNN loss being calculated as a sum of the losses from each time step as seen here: In Stanford's CS224N, I see loss calculated as an average of individual losses as seen ...
user avatar
  • 1
0 votes
1 answer
182 views

How to identify overfitting from LSTM plot, from the prediction on trained+unseen data

I am currently learning LSTM-RNN models and I have done some tests to see how they work. As in the most NN, overfitting and underfitting is a problem in ML. I have read articles such as this guy here: ...
user avatar
2 votes
1 answer
77 views

What model should I use to predict a time series like this?

This series is calculated from the difference of two day's stock index. I rescaled it using sklearn's StarndardScaler. It seems LSTM does not work well on this series.
user avatar
2 votes
0 answers
26 views

Many-to-many LSTM with large number of (correlated timestamps)

We are considering building a set of CRM tools (i.e. churn) using LSTM network for our online store. LSTM is chosen since it can handle naturally sequential nature of our (i.e. transactional) data, ...
user avatar
  • 21
1 vote
1 answer
45 views

What is a mixture of RNNs?

I am reading papers on different types of classification and prediction methods and keep coming across "Mixture of Recurrent Neural Networks" and "Mixture of Markov Chain Models". ...
user avatar
0 votes
0 answers
78 views

Many-to-many time-series prediction problem

I would like some advice on how to implement an RNN or LSTM for my problem. I am working in Keras Tensorflow. My data describes the moisture % histogram of a sample of material. There are 42 features ...
user avatar
0 votes
0 answers
16 views

Why does this code use MinMaxScaler to preprocess S&P 500 index data? [duplicate]

scaler = MinMaxScaler() sp500_scaled = pd.Series(scaler.fit_transform(sp500).squeeze(), index=sp500.index) sp500_scaled.describe() This ...
user avatar
2 votes
0 answers
42 views

Recurrent Neural Networks(RNNs): does Truncated Back-Propagation Through Time (TBPTT) make RNN less effective than their unfolded version?

RNNs are Turing-complete. However AFAIU, the usefulness of this feature (provided by the recurrent nature of RNNs) depends on the network weights. If the weights are shaped by TBPTT they should be ...
user avatar
  • 21
0 votes
1 answer
24 views

Does gradient clipping in a RNN help the network learn the long term dependencies?

So this was asked in one of the exams and I think that gradient clipping does help in learning long term dependencies in RNN but the answer provided to us was "Gradient clipping cannot help with ...
user avatar
2 votes
2 answers
287 views

How GRU solves vanishing gradient

I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative $\frac{\partial h_i}...
user avatar
  • 308
0 votes
0 answers
14 views

Multi-decoders may destroy VGG19 model property

A question about some statement that I didn't understood from the paper about stylization-based image colorization Stylization-Based Architecture for Fast Deep Exemplar Colorization (DOI: 10.1109/...
user avatar
1 vote
0 answers
26 views

How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
user avatar
  • 121
0 votes
1 answer
71 views

Component sizes in vanilla RNN

I would like to seek some clarifications on the dimensionalities of the components and weight parameters in a vanilla RNN model performing text classification for the next word. I will present my ...
user avatar
  • 308
3 votes
1 answer
84 views

Recurrent neural networks of unspecified size

Are there recurrent neural networks (RNNs) of variable size? From what I've seen, RNNs are usually built using several nodes (or layers), in a manner similar to unrolled hidden Markov models (HMMs); ...
user avatar
0 votes
0 answers
19 views

Fitting steady state of a recurrent neural network

Our model is $$\underline{\dot{\phi}} = (\mathbf{R}-\tau\mathbf{I} )\underline{\phi} + \mathbf{W}\underline{x}$$ where $\underline{\phi}\in\mathbb{R}^N$, $\mathbf{R}$ is a recurrent weight matrix ...
user avatar
  • 261
0 votes
0 answers
32 views

Time steps in RNN and LSTM

I am quite new to recurrent neural networks and how to use them for sequence classification. I was wondering if anyone could shed some light on how RNNs (specifically LSTMs) capture time. That is, can ...
user avatar
2 votes
1 answer
58 views

Jacobian of hidden state update in backpropagation through time

I am trying to understand the gradients of backpropagation through time for a simple recurrent neural network. In particular this one: https://arxiv.org/abs/1211.5063 (Section 1.1) (Also here: https://...
user avatar
0 votes
0 answers
25 views

How to make multivariate windowed time series data work for LSTM?

My dataset comprises 4-timestep sliding windows for multiple features and is structured as shown in the first image below. It is important to note that rows can be either (i) the next sliding window ...
user avatar
0 votes
0 answers
21 views

Different prediction performance when scaling data ALREADY between 0 and 1

I am fitting a LSTM neural network for time series forecasting on realized volatility data available here. I am only using one of the variables in the dataset, namely 'rv5' or the 5-minute realized ...
user avatar
1 vote
0 answers
41 views

How to encode variable-length unordered data

I am trying to learn an encoding of data, but I want to do so in a way that doesn't depend on the order of one of the inputs. The data is given by $\{(X_i,y_i)\}_{i=1}^{n}$ where $n$ is the number of ...
user avatar
  • 121
1 vote
0 answers
25 views

How to use LSTM for time series with the output as a sequence and not the future values

Currently, I have data of batteries and this data is recorded every second for different dates. So, I have data for 10 dates and the output that I need to predict is Zimg and Zreal, which are two ...
user avatar
0 votes
1 answer
43 views

Can Deep Recurrent Networks (Modelling of Sequence / Panel Data) handle feature vectors of different dimensions?

I am currently learning on sequence modeling with the coursera course from deeplearning.ai about RNN in general as well as the GRU and LSTM. However I am now one week in and still not sure, if these ...
user avatar
  • 61
0 votes
1 answer
66 views

Fitting a LSTM for stock price prediction using industry sector data

I am quite new to the theory of RNNs so please excuse me if the question is trivial. I am trying to fit a multivariate LSTM neural network to predict stock prices from a firm in the S&P 500 list. ...
user avatar
0 votes
0 answers
228 views

Transformer model training takes longer and results in lower train and validation loss

I have been making tests with Transformer model provided on Keras.io page, training for classification and seq2seq tasks in several datasets and compare Transformer to GRU/LSTM with almost same number ...
user avatar
0 votes
0 answers
18 views

Semi-supervised learning from longitudinal data

I wanted to get experts' opinion on the following scenario. I am analysing longitudinal biomarker data where serial measurements were collected over time and in the last time point we know exactly who ...
user avatar
  • 611
0 votes
0 answers
15 views

Recurrent neural networks with loss of data

I want to train a recurrent neural network (RNN) for making predictions of some data. I have 8 variable inputs and with them I have to make predictions of other two variables (outputs). I need RNN to ...
user avatar
  • 1
4 votes
2 answers
228 views

How to predict with a stateful LSTM the next values

I trained a RNN using LSTM cells. I would like to make a predictions for the next 14 days. I get results that are plausible but after reading various blogs I'm not so sure if I'm doing the right thing....
user avatar

1
2 3 4 5
13