# Tag Info

55

All RNNs have feedback loops in the recurrent layer. This lets them maintain information in 'memory' over time. But, it can be difficult to train standard RNNs to solve problems that require learning long-term temporal dependencies. This is because the gradient of the loss function decays exponentially with time (called the vanishing gradient problem). LSTM ...

37

The accepted answer focuses on the practical side of the question: it would require a lot of resources, if there parameters are not shared. However, the decision to share parameters in an RNN has been made when any serious computation was a problem (1980s according to wiki), so I believe it wasn't the main argument (though still valid). There are pure ...

35

LSTM shapes are tough so don't feel bad, I had to spend a couple days battling them myself: If you will be feeding data 1 character at a time your input shape should be (31,1) since your input has 31 timesteps, 1 character each. You will need to reshape your x_train from (1085420, 31) to (1085420, 31,1) which is easily done with this command : x_train=...

35

Maybe. But RNNs aren't. Transformers learn "pseudo-temporal" relationships; they lack the true recurrent gradient that RNNs have, and thus extract fundamentally different features. This paper, for example, shows that the standard transformers are difficult to optimize in reinforcement learning settings, especially in memory-intensive environments. ...

32

The vanishing gradient is best explained in the one-dimensional case. The multi-dimensional is more complicated but essentially analogous. You can review it in this excellent paper [1]. Assume we have a hidden state $h_t$ at time step $t$. If we make things simple and remove biases and inputs, we have $$h_t = \sigma(w h_{t-1}).$$ Then you can show that \...

32

I'll assume we're talking about recurrent neural nets (RNNs) that produce an output at every time step (if output is only available at the end of the sequence, it only makes sense to run backprop at the end). RNNs in this setting are often trained using truncated backpropagation through time (BPTT), operating sequentially on 'chunks' of a sequence. The ...

30

Your learning rate could be to big after the 25th epoch. This problem is easy to identify. You just need to set up a smaller value for your learning rate. If the problem related to your learning rate than NN should reach a lower error despite that it will go up again after a while. The main point is that the error rate will be lower in some point in time. ...

28

Attention is a method for aggregating a set of vectors $v_i$ into just one vector, often via a lookup vector $u$. Usually, $v_i$ is either the inputs to the model or the hidden states of previous time-steps, or the hidden states one level down (in the case of stacked LSTMs). The result is often called the context vector $c$, since it contains the context ...

28

It is funny that you ask now, since just today I came across a paper by Wang, Khabsa, and Ma (2020) To Pretrain or Not to Pretrain who show that if you have large enough training set, the difference in performance between huge, "SOTA" model (RoBERTa), and LSTMs is small for NLP task. There was another recent paper by Merity (2019) Single Headed ...

26

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights). Skip connections form conceptually a '...

25

I think that you are referring to vertically stacked LSTM layers (assuming the horizontal axes is the time axis. In that case the main reason for stacking LSTM is to allow for greater model complexity. In case of a simple feedforward net we stack layers to create a hierarchical feature representation of the input data to then use for some machine learning ...

25

The terminology is unfortunately inconsistent. num_units in TensorFlow is the number of hidden states, i.e. the dimension of $h_t$ in the equations you gave. Also, from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard9/tf.nn.rnn_cell.RNNCell.md : The definition of cell in this package ...

24

I found this just below the [samples, time_steps, features] you are concerned with. X = numpy.reshape(dataX, (len(dataX), seq_length, 1)) Samples - This is the len(dataX), or the amount of data points you have. Time steps - This is equivalent to the amount of time steps you run your recurrent neural network. If you want your network to have memory of 60 ...

15

Option 1 (adding an unknown word token) is how most people solve this problem. Option 2 (deleting the unknown words) is a bad idea because it transforms the sentence in a way that is not consistent with how the LSTM was trained. Another option that has recently been developed is to create a word embedding on-the-fly for each word using a convolutional ...

15

From my understanding, the BPC is just the average cross-entropy (used with log base 2). In the case of Alex Graves' papers, the aim of the model is to approximate the probability distribution of the next character given past characters. At each time step $t$, let's call this (approximate) distribution $\hat{P}_t$ and let $P_t$ be the true distribution. ...

15

Make the identity of the agent one of the features, and train on all data. Probably train on a mini-batch of eg 128 agents at a time: run through the time-series from start to finish for those 128 agents, then select a new mini-batch of agents. For each mini-batch, run a slice of say 50 timesteps, then backprop. Keep the end states from that slice, and run ...

14

I'd like to add some detail to the accepted answer, because I think it's a bit more nuanced and the nuance may not be obvious to someone first learning about RNNs. For the vanilla RNN, $$\frac{\partial h_{t'}}{\partial h_{t}} = \prod _{k=1} ^{t'-t} w \sigma'(w h_{t'-k})$$. For the LSTM, \frac{\partial s_{t'}}{\partial s_{t}} = \prod _{k=1} ^{t'-t} \...

14

Ironically the best Optimizers for LSTMs are themselves LSTMs: https://arxiv.org/abs/1606.04474 Learning to learn by gradient descent by gradient descent. The basic idea is to use a neural network (specifically here a LSTM network) to co-learn and teach the gradients of the original network. It's called meta learning. This method, while proposed by ...

14

Standard RNNs (Recurrent Neural Networks) suffer from vanishing and exploding gradient problems. LSTMs (Long Short Term Memory) deal with these problems by introducing new gates, such as input and forget gates, which allow for a better control over the gradient flow and enable better preservation of “long-range dependencies”. The long range dependency in ...

14

A very short answer: LSTM decouples cell state (typically denoted by c) and hidden layer/output (typically denoted by h), and only do additive updates to c, which makes memories in c more stable. Thus the gradient flows through c is kept and hard to vanish (therefore the overall gradient is hard to vanish). However, other paths may cause gradient explosion. ...

14

What you've outlined is probably the single most common error that machine learning researchers make when analyzing financial data: it's trivial to discover that a great predictor of tomorrow's price is today's price. The statistical term of art for this phenomenon is "non-stationarity." We have a number of questions about how to test for the ...

13

It totally depends on the nature of your data and the inner correlations, there is no rule of thumb. However, given that you have a large amount of data a 2-layer LSTM can model a large body of time series problems / benchmarks. Furthermore, you don't backpropagate-through-time to the whole series but usually to (200-300) last steps. To find the optimal ...

13

The 'shared weights' perspective comes from thinking about RNNs as feedforward networks unrolled across time. If the weights were different at each moment in time, this would just be a feedforward network. But, I suppose another way to think about it would be as an RNN whose weights are a time-varying function (and that could let you keep the ability to ...

13

I think there's some confusion here. The reason you have vanishing gradients in neural networks (with say, softmax) is wholly different from RNNs. With neural networks, you get vanishing gradients because most initial conditions make your outputs end up on either the far left or far right of your softmax layer, giving it a vanishingly small gradient. In ...

13

The purpose of the Rectified Linear Activation Function (or ReLU for short) is to allow the neural network to learn nonlinear dependencies. Specifically, the way this works is that ReLU will return input directly if the value is greater than 0. If less than 0, then 0.0 is simply returned. The idea is to allow the network to approximate a linear function ...

13

Our group recently built an LSTM model in a real world application. At first we had used other approaches, but then we decided to include features that were measurements taken over time, but of variable length - so for one person, we would have 15 measurements (of the same parameter) taken over a 3-month period, for another we would have 20 measurements over ...

12

In short, yes, you can get some measure of variable importances for RNN based models. I won't iterate through all of the listed suggestions in the question, but I will walk through an example of sensitivity analysis in depth. The data The input data for my RNN will be composed of a time-series with three features, $x_1$, $x_2$, $x_3$. Each feature will be ...

11

From {1}: While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that deep RNNs work better than shallower ones on some tasks. In particular, Sutskever et al (2014) report that a 4-layers deep architecture was crucial in achieving good machine-translation performance in an ...

11

In Keras LSTM(n) means "create an LSTM layer consisting of LSTM units. The following picture demonstrates what layer and unit (or neuron) are, and the rightmost image shows the internal structure of a single LSTM unit. The following picture shows how the whole LSTM layer operates. As we know an LSTM layer processes a sequence, i.e, \$\mathbb{x}_1, \dots, \...

10

The best place to start with LSTMs is the blog post of A. Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/. If you are using Torch7 (which I would strongly suggest) the source code is available at github https://github.com/karpathy/char-rnn. I would also try to alter your model a bit. I would use a many-to-one approach so that you input ...

Only top voted, non community-wiki answers of a minimum length are eligible