Questions tagged [recurrent-neural-network]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Can I use batchnorm in CNN + RNN, and where to place it exactly?

I have designed a following neural network that combines CNN, RNN and Dense layers. It aims to predict a positive or negative outcome for the time step t+1, given a ...
user avatar
  • 171
0 votes
1 answer
13 views

Learning stochastic pattern using RNN

I have a pattern of count time series of vehicle demand as shown below.The time series is generated as follows: Categorical Random Variable, x = {0,1,2} and p(x) = {0.6,0.3,0.1} low vehicles = 1 + x , ...
user avatar
0 votes
0 answers
5 views

Neural Network Feeding - Custom Information Extraction

I am trying to train a neural net. to extract specific information from text. I need to find entity information like attribute, dependency, etc. The text input will be like this: ...
user avatar
2 votes
1 answer
30 views

Why do large LMs use the transpose of the word embeddings matrix in the classification head?

All literature, guides and tutorials describing the construction of language models have used two separate matrices for the input and output projections: To project one-hot token IDs into hidden ...
user avatar
  • 1,036
0 votes
0 answers
8 views

referenced before assignment error in tensorflow raw_rnn [closed]

I am using the tensorflow's raw_rnn api to train a basicRnnCell. In the loop function I need to change the next_cell_state to last_Cell's state after every batch is processed. so in the 4th line of ...
user avatar
2 votes
1 answer
31 views

Learning a simple pattern with RNN

I am trying to make RNN in tensorflow capture a basic pattern in a simple time series in hours. I am trying to solve a bigger problem involving count time series of customer demand. The simple time ...
user avatar
0 votes
0 answers
6 views

RNN on count time series [duplicate]

I am trying to predict the following count time series using RNN. X axis is in hours. Y axis is customer demand. I have already tried using other methods like stochastic models in tscount in R. I am ...
user avatar
1 vote
1 answer
32 views

Sequential Binary Imbalanced data classification with LSTM

I'm building an LSTM sequential Binary Classification Model, the data is highly imbalanced like say Fraud detection case. After building an LSTM model on Sequential Vectorised data, I'm getting a very ...
user avatar
0 votes
1 answer
106 views

Interconnections between embeddings layer and LSTM layer

I'm trying to build a text classifier with keras using word embeddings (glove) and a RNN (in this case a LSTM) using keras. I searched in several sites and decided to start with this configuration: <...
user avatar
0 votes
0 answers
12 views

Different Series For Forecasting

I have a time series data for different programs and each program has its own sequence for applications.For example , program 2020/03 there are applicants who applied in year 2015,2016,2017 till the ...
user avatar
0 votes
0 answers
38 views

How does attention add expressive power to encoder-decoders?

I am learning about the attention mechanism for the first time, and the context in which it has been introduced (watching Lecture 8 of Stanford's CS224N) is that of language translation using seq2seq ...
user avatar
0 votes
0 answers
13 views

States in RNN: what if they are measured

A simple RNN will take the output and feed it back to a state variable. The next output is then the function of input and state. Now imagine I want to use the "state" variable in RNN as an ...
user avatar
0 votes
0 answers
54 views

RNN loss function spikes and slow decrease

I'm training a rnn model for classification (many to one) : ...
user avatar
  • 1
0 votes
0 answers
32 views

Understanding types of LSTM and their use cases

I'm currently considering to use RNN/LSTM for a predictive modelling project that involves time-variant points. From looking at the following types of LSTM/RNN (in the picture below), I want to try ...
user avatar
0 votes
0 answers
37 views

Time series forecasting for revenue forecasting?

I am currently working on a project where I have to forecast the revenue for (the duration of) projects within the organisation. The organisation has several departments that occupy themselves with a ...
user avatar
  • 1
0 votes
0 answers
22 views

Similar results between feedforward neural networks, recurrent neural network and LSTM for time series data - Is this standard?

Tl;dr: I have trained feedforward neural networks, recurrent neural networks and LSTM networks to predict behaviour of weather temperature. The results are almost all the same (see below). Is this ...
user avatar
  • 133
0 votes
0 answers
21 views

Question about understanding Weights of Keras LSTM model

I am implementing Federated Learning (FL) using Keras LSTM. Starting with the simple example where multiple models are trained at different clients. Each client shares their model weights with the ...
user avatar
  • 434
0 votes
0 answers
220 views

Evaluation for LSTM model

I have created a model for text generation using LSTM. I am having chess sequences learned, reporting only the pieces moved during the moves. So when I move a pawn on my game there will be "p&...
user avatar
0 votes
0 answers
25 views

Mini batches and loss in recurrent neural networks (RNNs)

Suppose that we have a sequence $\left\{x^{(k)}\right\}_{k = 1}^{N}$ and that we wish to use a RNN to predict the next element of the sequence given the previous elements of the sequence (e.g., a ...
user avatar
  • 101
0 votes
0 answers
15 views

How to predict a mathematical progression with keras

I try the following model for a many-to-many recurrent network: ...
user avatar
0 votes
0 answers
25 views

Vector to sequence RNNs: do they take a random initial "prompt"?

I am going through the Deep Learning book by Ian Goodfellow (here) and came by the architecture for a vector to sequence RNN (Figure 10.9). I am not sure I understand how this architecture works and ...
user avatar
1 vote
0 answers
50 views

How to train LSTM model for single time series prediction but with multiple dataframes? [closed]

I have to predict whether the shot will be disruptive or non-disruptive. What I have is time-series data of both the types of shots and I want the model to predict using initial period data whether it ...
user avatar
1 vote
1 answer
112 views

If I use transformers nn, I don’t need to use CTC loss and language models?

In the creation of an ASR (Automatic Speech Recognition) using transformers neural networks, we don’t need to use a CTC loss and language models, do we? I’ve seen that RNNs do need to use language ...
user avatar
3 votes
0 answers
46 views

Stacking recurrent neural network layers horizontally vs vertically

In the top answer on this post: What are the advantages of stacking multiple LSTMs? the idea of stacking LSTMs vertically is distinguished from stacking them horizontally. I quite don't understand ...
user avatar
  • 31
0 votes
0 answers
17 views

Is the recursive form in a transformer efficient for long sequences?

I am reading the paper Transformers are RNNs:Fast Autoregressive Transformers with Linear Attention, where the authors propose the linear transformer, which seems to be very efficient in practice. Yet,...
user avatar
0 votes
0 answers
18 views

Trying to predict the next ignition off for a vehicle

Overview: I have a dataset that captures 6 months of data for 1000 cars. Each car is represented by it's unique identifier. The data captures the exact timestamp when the car was turned ON and turned ...
user avatar
  • 1
0 votes
0 answers
15 views

Evolving action and selection for automated agents

I have been reading this paper 1, and I came through this graph, which I had a hard time understanding. Basically, we have an environment with food and poison, and water, our agent has three sensors, ...
user avatar
1 vote
1 answer
90 views

Backpropagation through time: dimensions in the chain rule

Take the first step $t=1$ for simplicity, \begin{align} h_1 &= \text{tanh}(W_{hx}^T e_1 + W_{hh}^T h_0) \end{align} Usually it is written in convention: \begin{align} \frac{\partial L_1}{\...
user avatar
  • 308
0 votes
1 answer
52 views

Sequential recommendation: how to effective encoding output item?

Now I am learning about sequential recommendation - session based recommendation. I have understood that User-item interactions may be viewed as sequential action (first I clicked item A, then click ...
user avatar
  • 150
1 vote
1 answer
212 views

ANN vs RNN for time-series prediction?

The basic argument against ANN for time-series is that there is no conception of time for the inputs. However, it is obvious that one can feed multiple time-steps into an ANN by using various lag ...
user avatar
0 votes
2 answers
36 views

What degree of difference does validation and training loss need to have to be called overfit?

I've trained an LSTM network to predict time series data however i'm quite new to LSTMs and am unsure if the model has overfit. I know that an increasing validation loss relative to a decreasing ...
user avatar
2 votes
1 answer
191 views

Validity of basic train - test - split for a time series using a RNN

I am trying to determine if a simple train-test-split is valid for a time series if I use a Recurrent Neural Network (LSTM). Lets say I have samples (x) which consist of 2 days values (time steps) ...
user avatar
  • 23
0 votes
1 answer
54 views

What are the transition functions for RNNs

From what I understand, the hidden states of RNNs are equivalent to the deterministic probability distribution over hidden states in for example a Hidden Markov Model. Thus, just as probabilistic ...
user avatar
0 votes
0 answers
13 views

Calculating RNN loss (for a SINGLE example) as a sum of individual time step losses VS. an average of individual time step losses [duplicate]

In Andrew Ng's course, I see RNN loss being calculated as a sum of the losses from each time step as seen here: In Stanford's CS224N, I see loss calculated as an average of individual losses as seen ...
user avatar
  • 1
0 votes
1 answer
330 views

How to identify overfitting from LSTM plot, from the prediction on trained+unseen data

I am currently learning LSTM-RNN models and I have done some tests to see how they work. As in the most NN, overfitting and underfitting is a problem in ML. I have read articles such as this guy here: ...
user avatar
2 votes
1 answer
87 views

What model should I use to predict a time series like this?

This series is calculated from the difference of two day's stock index. I rescaled it using sklearn's StarndardScaler. It seems LSTM does not work well on this series.
user avatar
2 votes
0 answers
52 views

Many-to-many LSTM with large number of (correlated timestamps)

We are considering building a set of CRM tools (i.e. churn) using LSTM network for our online store. LSTM is chosen since it can handle naturally sequential nature of our (i.e. transactional) data, ...
user avatar
  • 21
1 vote
1 answer
49 views

What is a mixture of RNNs?

I am reading papers on different types of classification and prediction methods and keep coming across "Mixture of Recurrent Neural Networks" and "Mixture of Markov Chain Models". ...
user avatar
0 votes
0 answers
200 views

Many-to-many time-series prediction problem

I would like some advice on how to implement an RNN or LSTM for my problem. I am working in Keras Tensorflow. My data describes the moisture % histogram of a sample of material. There are 42 features ...
user avatar
0 votes
0 answers
17 views

Why does this code use MinMaxScaler to preprocess S&P 500 index data? [duplicate]

scaler = MinMaxScaler() sp500_scaled = pd.Series(scaler.fit_transform(sp500).squeeze(), index=sp500.index) sp500_scaled.describe() This ...
user avatar
2 votes
0 answers
60 views

Recurrent Neural Networks(RNNs): does Truncated Back-Propagation Through Time (TBPTT) make RNN less effective than their unfolded version?

RNNs are Turing-complete. However AFAIU, the usefulness of this feature (provided by the recurrent nature of RNNs) depends on the network weights. If the weights are shaped by TBPTT they should be ...
user avatar
  • 21
0 votes
1 answer
53 views

Does gradient clipping in a RNN help the network learn the long term dependencies?

So this was asked in one of the exams and I think that gradient clipping does help in learning long term dependencies in RNN but the answer provided to us was "Gradient clipping cannot help with ...
user avatar
2 votes
2 answers
517 views

How GRU solves vanishing gradient

I am learning the GRU model in deep learning and reading this article where details of BPTT are explained. Towards the end the author explained the values of the partial derivative $\frac{\partial h_i}...
user avatar
  • 308
0 votes
0 answers
18 views

Multi-decoders may destroy VGG19 model property

A question about some statement that I didn't understood from the paper about stylization-based image colorization Stylization-Based Architecture for Fast Deep Exemplar Colorization (DOI: 10.1109/...
user avatar
1 vote
0 answers
28 views

How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
user avatar
  • 121
0 votes
1 answer
102 views

Component sizes in vanilla RNN

I would like to seek some clarifications on the dimensionalities of the components and weight parameters in a vanilla RNN model performing text classification for the next word. I will present my ...
user avatar
  • 308
3 votes
1 answer
85 views

Recurrent neural networks of unspecified size

Are there recurrent neural networks (RNNs) of variable size? From what I've seen, RNNs are usually built using several nodes (or layers), in a manner similar to unrolled hidden Markov models (HMMs); ...
user avatar
0 votes
0 answers
31 views

Fitting steady state of a recurrent neural network

Our model is $$\underline{\dot{\phi}} = (\mathbf{R}-\tau\mathbf{I} )\underline{\phi} + \mathbf{W}\underline{x}$$ where $\underline{\phi}\in\mathbb{R}^N$, $\mathbf{R}$ is a recurrent weight matrix ...
user avatar
  • 261
0 votes
0 answers
75 views

Time steps in RNN and LSTM

I am quite new to recurrent neural networks and how to use them for sequence classification. I was wondering if anyone could shed some light on how RNNs (specifically LSTMs) capture time. That is, can ...
user avatar
2 votes
1 answer
67 views

Jacobian of hidden state update in backpropagation through time

I am trying to understand the gradients of backpropagation through time for a simple recurrent neural network. In particular this one: https://arxiv.org/abs/1211.5063 (Section 1.1) (Also here: https://...
user avatar

1
2 3 4 5
13