Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
0
votes
1answer
13 views

Generating text using LSTM given condition vector

I know that you can use an RNN to generate text given the first few letters ...
1
vote
1answer
20 views

Can a RNN make a joint prediction and if so, is this implementable in Python software like Tensorflow or Pytorch?

I was wondering, can a RNN make a joint prediction (or predict two outputs at once)? For example, let's say I want to predict tomorrow's weather. I want to predict both temperature and humidity as a ...
2
votes
1answer
15 views

Information loss in recurrent neural networks?

I am reading Deep Learning by Goodfellow and he seems to imply that the latter structure for a RNN will lead to information loss over the first structure. In both of the pictures, we have a RNN. x is ...
0
votes
1answer
21 views

How to model properly sequential data when the output has to be used as part of the next input? Model off completely when it makes single mistake

I have time series data and am fitting a (LSTM) neural network. The time series data include let's say a brain wave (var1) as well as the previous state (prev_state) and I want to predict a state (...
0
votes
0answers
17 views

Why do people use tanh more often than ReLU in vanilla recurrent neural networks?

For instance, the default activation function of tf.keras.layers.SimpleRNN is tanh. My doubt is because tanh activation functions may also cause (like sigmoids) the ...
2
votes
0answers
60 views

Basic RNN sequence classifier diagram?

I'd like to build an RNN in numpy from scratch to really get come comfortable with backpropagation through time (BPTT.) In the below diagram and LaTeX, I show two neurons, each with a non-linearity, N(...
0
votes
0answers
23 views

How to model a probability that event will happen in next XY time units?

I want to do a binary prediction for time series data. I have data on weekly basis, i.e. each week I get a multivariate observation for each subject. For each subject there is a binary label at each ...
0
votes
0answers
9 views

Insufficient n > p for forecasting with an RNN

What is the best approach to handle insufficient sample size when forecasting for multiple sequences simultaneously with an RNN? My training set has n=956 (time points) and p=262 (sequences). I'm ...
1
vote
0answers
14 views

Why do we fit Recurrent Neural Networks with backprop instead of message passing/expectation propagation?--as with hidden markov models

The form of a Recurrent Neural Network (RNN) seems to resemble that of a hidden markov model. With a hidden markov model we have transitions between discrete states, as well as an emission variable ...
1
vote
0answers
10 views

Approaches for deep learning of interdependent sequences?

On the net there are tons of examples of recurrent neural network usage to "learn" sequences like music, texts, or other kind of ordered patterns, but in economics often such sequences are ...
1
vote
0answers
8 views

LSTM high accuracy but poor generation performance

I'm writing a LSTM model for generating music (in particular drums). My model is based on these 2 models: LSTM text generator LSTM drum generator The model seems to work fine, it trains and I can ...
0
votes
0answers
8 views

Why are RNNs used in some computer vision problems?

I am learning computer vision. When I was going through implementations of various computer vision projects, some OCR problems used GRU or LSTM, while some did not. I understand that RNNs are used ...
0
votes
0answers
13 views

BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?

I'm reading the BERT paper and jalammar's illustrative guide for BERT. I don't understand 2 things about the method's crux - the masked language model: why does masking requires us to sample (take ...
0
votes
0answers
8 views

seq2seq models with attention: which components are truly duplicated per word of the source sentence and which are just unrolled?

I'm reading jalammar's description of seq2seq with attention and tensorflow impl of the same. Let's say that we're trying to translate a sequence (sentence) of 4 words ...
0
votes
0answers
10 views

Predicting multilinear regresssion coefficients using an RNN

If you had a multlinear regression problem (y = $a_{1}x_1$ + $a_{2}x_{2}$ + $a_{3}x_{3}$+ ... + $a_{n}x_{n}$+ b, want to predict the coefficients, $a_{1}$, $a_{2}$, ..., $a_{n}$, and b), could you use ...
0
votes
0answers
16 views

Why in gated recurrent unit gates are controlled by only one layer perceptrons?

Why don't I see a GRU anywhere with more than one layer of perceptrons inside, it's pretty obvious to try to put more layers in there, but I don't see anyone doing that
0
votes
0answers
10 views

How to combine categorical and numeric data in pytorch [duplicate]

I modelling a problem which has categorical features which I have one hot encoded. I want to use categorical feature along with numeric feature as input. Example, purchase category and purchase time. ...
2
votes
1answer
54 views

Weird behaviour in toy RNN (Keras, LSTM)

I'm trying to learn more about RNNs and I'm tackling a toy problem. I'm generating some data that has a pattern, two 1s followed by three 0s which keeps repeating infinitely without any noise. So my ...
1
vote
2answers
29 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
1
vote
1answer
47 views

Are RNNs Markovian?

On the one hand, one can argue that they are since "the hidden layer is simply [derived from] the last hidden state and current input". On the other hand, the whole point of RNNs is that &...
0
votes
0answers
14 views

Training an LSTM on multiple distinct batches of time series data

I am running a time series simulation on an electricity power grid simulation package and I want to use this data to train an LSTM to predict the stability of the grid over a given time interval. My ...
0
votes
0answers
22 views

Use deep learning to predict the continuation of one time series given another

I have this problem I'm trying to solve.. I have 2 highly correlated time series (lets call them $A$ & $B$), however past a certain date I only have data for A. I would like to use $A$ to predict ...
1
vote
0answers
4 views

Dealing with missing timesteps in RNNs when the time dimension matters

One way I've found to deal with missing timesteps for an RNN is to process the instances with a batch size = 1. However, in my case, the time dimension matters. To understand what I mean, here's a ...
0
votes
0answers
12 views

Should LSTM data be a sequence?

let me explain what I want to do, I want to predict the trend of the price of something (1 if it increases in the next hour and 0 otherwise). I have gathered tweets about that and grouped them in ...
0
votes
0answers
23 views

How exactly does conv1d filter work when operating on a sequence of characters?

I understand convolution filters when applied to an image (e.g. an 224x224 image with 3 in-channels transformed by 56 total filters of 5x5 conv to a 224x224 image with 56 out-channels). The key is ...
0
votes
1answer
66 views

When computing parameters, why is dimensions of hidden-output state of an LSTM-cell assumed same as the number of LSTM-cell?

I was trying to figure out how to estimate the number of parameters in an LSTM layer. What is the relationship of number of parameters with the num lstm-cells, input-dimension, and hidden output-state ...
0
votes
0answers
12 views

Number of bidirectional LSTMs in encoder-decoder model must equal the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
0
votes
0answers
17 views

Can RNNs handle multi-dimensional inputs?

I'm a bit confused. I'm trying to build an RNN without much guidance by following PyTorch docs and RNN lectures. Are they able to handle multi-dimensional inputs or is that something for LSTMs? I'm ...
0
votes
1answer
39 views

LSTM network window size selection and effect

When working with an LSTM network in Keras. The first layer has the input_shape parameter show below. model.add(LSTM(50, input_shape=(window_size, num_features), return_sequences=True)) I don't ...
0
votes
0answers
20 views

Model architecture approaches for event prediction at different timestamps

I would like to model a user event outcome (currently its binary). the data I have is aggregated user activity and static user data. here is an example of what the data looks like for clarity: ...
1
vote
0answers
18 views

State Design in modern Elevator [closed]

I like to implement one language model for elevator operations. To do this I worked out on how elevator works, its time, counterweight aspect, etc. Next, I found it operates mainly by a priority ...
0
votes
0answers
9 views

introducing lag variables versus using RNNs in time series prediction

Just wondering, is there a fundamental difference between introducing lagged IVs during data preparation and then using standard statistical/machine learning models versus the uses of RNNs such as ...
2
votes
1answer
71 views

Backpropagation through time for RNN: how to deal with recursively defined gradient updates?

A simplified RNN architecture basically involves the following update \begin{equation} \begin{cases} h_t & = \phi(w h_{t-1} + v x_t )\\ \hat y_t & = \theta(h_t ) \end{cases} \end{...
1
vote
2answers
147 views

What is a multilayer LSTM?

Apologies, this question is quite long. I am trying to implement a paper on optimising the working of multilayer LSTM. The optimisation process works as follows: First I wrote a sequential code for ...
0
votes
1answer
28 views

Are neural networks smart enough to overcome unbalanced data sets?

From what I have read, many models have issues with unbalanced data sets in classification problems. Are neural networks smart enough to overcome this flaw or should I still look into creating a ...
0
votes
0answers
11 views

Advice for modeling a data set with several different data types

Ok so I have a project with a somewhat complex dataset and I'm looking for some guidance. The project is basically to try to predict NBA player performance (points scored) each game, for each player. ...
0
votes
1answer
29 views

IID Assumption in Sequential Supervised Learning

I have seen two posts about this before (Are RNNs inherently flawed? Supervised Learning assumes IID data but sequential data is not IID, Realistically, does the i.i.d. assumption hold for the vast ...
0
votes
0answers
4 views

Creating predictions from RNN model built on scaled data

As the title says, how would I create predictions from a RNN model trained on scaled (-1,1) data, especially when the test data/real world data isn't in the range of the non scaled data?
0
votes
0answers
12 views

FCNN with stacking/sequencing data rows vs RNN

I'm training a binary classifier on a time-series data set. I want to know whats the differences between these two scenarios and could the second one perform better in any cases? 1 - using RNN or ...
0
votes
1answer
272 views

Should the Dropout vs. Recurrent Dropout Arguments Be the Same in Keras?

I'm learning about recurrent neural networks right now, and am in chapter 6 of Deep Learning with Python by Francois Chollet. In the chapter it's discussing using dropout in recurrent layers. I ...
0
votes
0answers
12 views

Independence Assumption in Time Series Forecasting using Recurrent Neural Network

In Neural Network and Recurrent Neural Network, we need to have independence between the observation because these architectures rely on some of independence assumptions. But if we have one time ...
1
vote
0answers
12 views

Normalization functions in RNN LSTM

I've read somewhere that the tanh function was introduced in order to combat the problem of vanishing exploding gradient. However not many sources explain why ...
0
votes
1answer
40 views

Transformers for embedding sequences as fixed-length vectors

An encoder decoder network of recurrent neural networks can be used to learn the identity function over some set of sequences. If you do this without attention, the output of the decoder can be ...
0
votes
0answers
9 views

Generating Fixed Length Sequence with RNNs

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
0
votes
1answer
40 views

What are the advantages of combining BiLSTM and CRF?

BiLSTM-CRF is a common model for sequence tagging (POS tagging, NER, ect.). What are the advantages of combining BiLSTM and CRF? What is the role of each one of the parts in this combination?
2
votes
1answer
27 views

Should the samples in the input data into an RNN always be temporally ordered?

From what I know, if the training set shape is [100, 500, 20], it represents 100 samples, each sample being 500 timeseries and each timeseries having 20 features. I'm wondering if I'm passing for ...
0
votes
0answers
31 views

RNN test error calculation: teacher forcing or prediction data?

RNN training is mostly done by a teacher forcing method: [Training] y_{pred}^{i+1} = RNN(y_{real}^{i}) [Training error] Error_{train} = abs(y_{real}^{i+1} - y_{pred}^{i+1}) However, RNN applications ...
0
votes
1answer
37 views

Time Series in Neural Network?

We know that Neural Network are usually designed for independent observations. This assumptions comes from the loss function where it is more easy to compute it when we have independent observations. ...
1
vote
1answer
21 views

RNN exploding/vanishing gradient, hidden cell formulation

I was reading a RNN paper that discuss vanishing/exploding gradient: http://proceedings.mlr.press/v28/pascanu13.pdf and when they present Eq. 2, they assume that $$ x_t = W_{rec} \sigma(x_{t-1}) + W_{...
0
votes
1answer
32 views

Are RNNs in Keras dynamic length?

Sorry if this seems like a basic question. From my understanding, one of the advantages of sequence models like RNNs is that they can handle variable length input sequences. For example, if I'm ...

1
2 3 4 5
10