Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Why does PyTorch use two separate networks in the RNN tutorial? [closed]

In this tutorial for a simple RNN, PyTorch uses two indipendent networks, i2h and i2o, producing independently the hidden state ...
2
votes
0answers
16 views

Many-to-many LSTM with large number of (correlated timestamps)

We are considering building a set of CRM tools (i.e. churn) using LSTM network for our online store. LSTM is chosen since it can handle naturally sequential nature of our (i.e. transactional) data, ...
1
vote
1answer
31 views

What is a mixture of RNNs?

I am reading papers on different types of classification and prediction methods and keep coming across "Mixture of Recurrent Neural Networks" and "Mixture of Markov Chain Models". ...
0
votes
0answers
20 views

Many-to-many time-series prediction problem

I would like some advice on how to implement an RNN or LSTM for my problem. I am working in Keras Tensorflow. My data describes the moisture % histogram of a sample of material. There are 42 features ...
0
votes
0answers
16 views

Why does this code use MinMaxScaler to preprocess S&P 500 index data? [duplicate]

scaler = MinMaxScaler() sp500_scaled = pd.Series(scaler.fit_transform(sp500).squeeze(), index=sp500.index) sp500_scaled.describe() This ...
2
votes
0answers
18 views

Recurrent Neural Networks(RNNs): does Truncated Back-Propagation Through Time (TBPTT) make RNN less effective than their unfolded version?

RNNs are Turing-complete. However AFAIU, the usefulness of this feature (provided by the recurrent nature of RNNs) depends on the network weights. If the weights are shaped by TBPTT they should be ...
0
votes
1answer
18 views

Does gradient clipping in a RNN help the network learn the long term dependencies?

So this was asked in one of the exams and I think that gradient clipping does help in learning long term dependencies in RNN but the answer provided to us was "Gradient clipping cannot help with ...
0
votes
0answers
22 views

Analysing the behaviour of loss gradients when fitting keras models on text data

I have fitted SimpleRNN and LSTM separately on this comment text data for binary sentimental classification. Each row is a text sequence and the target is a 0-1 score. I have used GloVe embeddings as ...
0
votes
0answers
11 views

Multi-decoders may destroy VGG19 model property

A question about some statement that I didn't understood from the paper about stylization-based image colorization Stylization-Based Architecture for Fast Deep Exemplar Colorization (DOI: 10.1109/...
1
vote
0answers
21 views

How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
0
votes
1answer
43 views

Component sizes in vanilla RNN

I would like to seek some clarifications on the dimensionalities of the components and weight parameters in a vanilla RNN model performing text classification for the next word. I will present my ...
3
votes
1answer
79 views

Recurrent neural networks of unspecified size

Are there recurrent neural networks (RNNs) of variable size? From what I've seen, RNNs are usually built using several nodes (or layers), in a manner similar to unrolled hidden Markov models (HMMs); ...
0
votes
0answers
11 views

Fitting steady state of a recurrent neural network

Our model is $$\underline{\dot{\phi}} = (\mathbf{R}-\tau\mathbf{I} )\underline{\phi} + \mathbf{W}\underline{x}$$ where $\underline{\phi}\in\mathbb{R}^N$, $\mathbf{R}$ is a recurrent weight matrix ...
0
votes
0answers
16 views

Time steps in RNN and LSTM

I am quite new to recurrent neural networks and how to use them for sequence classification. I was wondering if anyone could shed some light on how RNNs (specifically LSTMs) capture time. That is, can ...
2
votes
1answer
48 views

Jacobian of hidden state update in backpropagation through time

I am trying to understand the gradients of backpropagation through time for a simple recurrent neural network. In particular this one: https://arxiv.org/abs/1211.5063 (Section 1.1) (Also here: https://...
0
votes
0answers
18 views

How to make multivariate windowed time series data work for LSTM?

My dataset comprises 4-timestep sliding windows for multiple features and is structured as shown in the first image below. It is important to note that rows can be either (i) the next sliding window ...
0
votes
0answers
19 views

Different prediction performance when scaling data ALREADY between 0 and 1

I am fitting a LSTM neural network for time series forecasting on realized volatility data available here. I am only using one of the variables in the dataset, namely 'rv5' or the 5-minute realized ...
1
vote
0answers
18 views

How to encode variable-length unordered data

I am trying to learn an encoding of data, but I want to do so in a way that doesn't depend on the order of one of the inputs. The data is given by $\{(X_i,y_i)\}_{i=1}^{n}$ where $n$ is the number of ...
0
votes
0answers
9 views

Understanding the subscripts in RNN components

I am learning the RNN model structure after familiarising with MLP structure. I am having some trouble understanding the dimensionality in the components of an RNN. Onn page 19 of Graves' notes, he ...
1
vote
0answers
21 views

How to use LSTM for time series with the output as a sequence and not the future values

Currently, I have data of batteries and this data is recorded every second for different dates. So, I have data for 10 dates and the output that I need to predict is Zimg and Zreal, which are two ...
0
votes
1answer
39 views

Can Deep Recurrent Networks (Modelling of Sequence / Panel Data) handle feature vectors of different dimensions?

I am currently learning on sequence modeling with the coursera course from deeplearning.ai about RNN in general as well as the GRU and LSTM. However I am now one week in and still not sure, if these ...
0
votes
1answer
57 views

Fitting a LSTM for stock price prediction using industry sector data

I am quite new to the theory of RNNs so please excuse me if the question is trivial. I am trying to fit a multivariate LSTM neural network to predict stock prices from a firm in the S&P 500 list. ...
0
votes
0answers
15 views

Transformer model training takes longer and results in lower train and validation loss

I have been making tests with Transformer model provided on Keras.io page, training for classification and seq2seq tasks in several datasets and compare Transformer to GRU/LSTM with almost same number ...
0
votes
0answers
13 views

Semi-supervised learning from longitudinal data

I wanted to get experts' opinion on the following scenario. I am analysing longitudinal biomarker data where serial measurements were collected over time and in the last time point we know exactly who ...
0
votes
0answers
14 views

Recurrent neural networks with loss of data

I want to train a recurrent neural network (RNN) for making predictions of some data. I have 8 variable inputs and with them I have to make predictions of other two variables (outputs). I need RNN to ...
4
votes
2answers
118 views

How to predict with a stateful LSTM the next values

I trained a RNN using LSTM cells. I would like to make a predictions for the next 14 days. I get results that are plausible but after reading various blogs I'm not so sure if I'm doing the right thing....
1
vote
1answer
160 views

Difference between token embedding and character embedding in ELMo model

I am learning about a famous NLP model called ELMo. In the explanations, they talk about two types of embeddings. 1) character representations 2) token representations. Why is there a need to consider ...
0
votes
0answers
18 views

Getting better results on LSTM (Tensorflow) shaping timesteps into features (?)

I am building multiple models for several different environments (similar applications) with LSTMs on Tensorflow. I know and I have seen people commenting on how inappropriate it is to build LSTMs ...
1
vote
1answer
19 views

When should RNNs be used, seeing as they take much longer to train?

RNNs seem to take much longer to train in most if not all cases. I assume this is because the number of operations involved in training an RNN scales not only with the number of examples being fed ...
1
vote
0answers
13 views

In LSTM texts, sometimes the current input and previous hidden state are concatenated, then multiplied by a weight. Sometimes, the other way. Why?

In the first case, the input and previous hidden state are concatenated then multiplied by a weight matrix, resulting in four matrix multiplications. e.g. https://bond-kirill-alexandrovich.medium.com/...
0
votes
0answers
13 views

Why loss function of my LSTM model drops a lot after training using the first 100 batches and then almost doesn't decrease?

I am training a LSTM model with one layer and 32 hidden neurons in tensorflow using tf.keras.layers.LSTMCell. The problem is a binary classification and the loss ...
0
votes
0answers
56 views

Is GRU the minimum/simplest RNN to prevent vanishing or exploding?

I learned from this answer and this post that the forget gate in LSTM controls which information to vanish and which not, but I wonder if the LSTM or GRU is the minimum/simplest RNN to accomplish that(...
0
votes
0answers
12 views

Model predicting extremely different output for examples outside the dataset [duplicate]

I'm training a neural machine translator using the encoder-decoder approach and I've got around 95% validation accuracy and 0.4 validation loss. The model is translating correctly for most of the ...
0
votes
0answers
23 views

Recommendation for prediction of multivariate time series

I am looking for a recommendation (concrete package/framework/approach etc., preferably in Python/Keras or R as that is where I have experience) for predicting multivariate time series. I do have ...
0
votes
0answers
15 views

Is this transformation to the input of an RNN "valid"?

I'm reading through some pytorch code published as part of a research paper. The input data is of the shape (batch_size, number_of_time_steps, number_of_predictor_variables, height, width). The code ...
1
vote
1answer
64 views

Understanding number of rnn units in RNN networks

I am trying to learn about recurrent neural networks from here. There are rnn_units = 1024 in the model and each batch contains ...
1
vote
0answers
60 views

Predicting sequence of sets

I have dataset which contains users and their market baskets (itemsets). The goal is to predict next basket. Can I predict next itemset in sequence of sets (itemsets) using RNN or another NN? Like ...
0
votes
0answers
26 views

Why do the decoders in a transformer only receive the output from the final encoder?

As the title suggests, in an $N$ layer transformer architecture each $Encoder_{i+1}$ operates on the state output from $Encoder_i$, but all $Decoder_i$ only receive the state from $Encoder_N$. This ...
0
votes
0answers
12 views

Machine Learning Model Not Learning [duplicate]

I'm trying to build a LSTM model that takes in 150 consecutive candlesticks' open, high, low, close and EMA indicator values (150, 5), then predicts whether price moves 20 points up or down first. <...
0
votes
1answer
19 views

How to verify if a graphical model has the markov property?

If I draw the computational graph of an HMM and an RNN from an architectural point of view they look very similar. The main difference is that an RNN gets some input $x$ and the HMM only operates on ...
0
votes
1answer
78 views

How do I prepare clinical data for multivariate time series analysis using LSTM or RNNs?

I am trying to predict the progression of disease using certain clinical data (time series data) and covariates (such as age, sex, race etc.). I am aware of the existence of mainstream machine ...
1
vote
0answers
32 views

Imputation Methods for Multiple Variables

I am training Recurrent Neural Network models (including LSTMs) on a dataset that includes 6-10 variables. Each variable is a properly formatted numerical measurement (ie: length, pressure, ...
2
votes
0answers
37 views

How to preprocess binary labeled sequences of ordered in time data points to fit into RNN?

I am observing some repeated events which are guaranteed to give a binary outcome on termination and I am doing some data preparation so I can fit it into a model. Each event has its own properties ...
0
votes
0answers
17 views

How do LSTMs understand correlations between features?

Let's say I want to create a model that predicts the stock prices of 2 stocks, for each stock I have additional information such as the volume. So the data I have would be Stock1Price, Stock1Volume, ...
0
votes
0answers
21 views

Can't an LSTM count the number of cycles of the input?

Can an LSTM count number of cycles? Or, equivalently predict a linear trend from a sinusoidal input? I have tried to train an LSTM network on a problem that I supposed would be easy for it. The ...
0
votes
1answer
30 views

Selecting a label smoothing factor for seq2seq NMT with a massive imbalanced vocabulary

I'm training a seq2seq RNN with a vocabulary of 8192 words. This means that the typical categorical cross entropy label smoothing factor suggested in papers like 'Attention is all you need' of $0.1$ ...
0
votes
1answer
33 views

Run-length encoding on the input sequence of a RNN

I have a dataset consisting of sequences of item embeddings a, b, c, ..., in which the length of consecutive runs of an item may be large (comparative to the ...
1
vote
1answer
32 views

multiple likely ys for one instance of x: word prediction with LSTM

I have a ML project that is about predicting (suggesting) the next word based on the last n words, using LSTM. The output is a softmax dense layer the size of the vocabulary that shows the probability ...
0
votes
0answers
11 views

In sequence to sequence RNN(Eg: English to Spanish translation), why gradients are summed in reverse order of time?

In sequence to sequence RNN(Eg: English to Spanish translation), multiple inputs & outputs are there(or multiple timesteps, say T). Below equations are used to predict output at each timestep: ...
1
vote
1answer
31 views

Are LSTMs just RNNs that use a different activation function?

I read online that LSTMs are just RNNs that calculate neuron activations differently. I am not clear on the terminology though. Do we actually call these activations, because they refer not to the ...

1
2 3 4 5
12