We’re rewarding the question askers & reputations are being recalculated! Read more.

# Tag Info

9

Yes, cell output equals to the hidden state. In case of LSTM, it's the short-term part of the tuple (second element of LSTMStateTuple), as can be seen in this picture: But for tf.nn.dynamic_rnn, the returned state may be different when the sequence is shorter (sequence_length argument). Take a look at this example: n_steps = 2 n_inputs = 3 n_neurons = 5 X ...

9

According to Rahul Dey and Fathi M. Salem, "Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks": ... the total number of parameters in the GRU RNN equals $3 \times (n^2 + nm + n)$. where $m$ is the input dimension and $n$ is the output dimension. This is due to the fact that there are three sets of operations requiring weight matrices of these ...

6

Now, the first number when calling "LSTM(...)" UNITS means that each LSTM cell has a history of 5 values, so there are 4 more LSTM-cells behind a single LSTM cell. When we unroll the LSTM we can see 5 LSTM-cells, right? Not really. "Units" means how many neurons (or cells) your network will have. This network will then be copied (unfolded) t number of times,...

5

Yes, the whole basis of AE training is that they try to learn a mapping function that maps similar inputs into nearby positions in a lower-dimensional space. If the AE has been trained properly (which you can't be sure of), then there is a high possibility that similar inputs would get nearby mappings. But that isn't by any means guaranteed.

4

Yes, the output of both the LSTM's forget gate and the GRU's remember gate are vectors with dimension equal to the LSTM's cell memory and GRU's hidden state, respectively, and pointwise values between 0 and 1. They act by multiplying pointwise by the previous LSTM cell memory or the previous GRU hidden state, respectively. (Note, unlike the LSTM, the GRU ...

3

This article is a good place to start. "Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network" by Alex Sherstinsky Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles,...

3

The universal approximation theorem asserts that there exsits a feed forward neural network to approximate a function. From the wiki page "however, it does not touch upon the algorithmic learnability of those parameters". In other words, although the universal approximation guarantees the existance of such a neural network, we cannot guarantee that there is ...

2

Here are my suggestion to pinpoint the issue: 1) Look at training learning curve: How is the learning curve on train set? Does it learn the training set? If not, first work on that to make sure you can over fit on the training set. 2) Check your data to make sure there is no NaN in it (training, validation, test) 3) Check the gradients and the weights to ...

2

Possible copy of https://stackoverflow.com/questions/36817596/get-last-output-of-dynamic-rnn-in-tensorflow/49705930#49705930 Anyway let's go ahead with the answer. This code snip might help understand what's really being returned by the dynamic_rnn layer => Tuple of (outputs, final_output_state). So for an input with max sequence length of T time ...

1

So if I use a bi-directional LSTM/GRU layer over Fasttext representations of words, will it be the same? Sep. 27, 2018 the authors published a follow on work to ELMo, "Dissecting Contextual Word Embeddings: Architecture and Representation". They test whether a bidirectional language model (biLM) can be replicated using Transformers, CNNs, and deeper LSTMs....

1

What you are looking for is the "many to many" architecture, where you input one vector at a time and then it predicts as many vectors as you need. Courtesy of Andrej Karpathy And 5000 shouldn't be a problem for deep recurrent networks using advanced (LSTM/GRU) gates. Google WaveNet generates raw audio samples, which are much longer than 5000.

1

I would initially just focus on the response variable and consider an ensemble forecast where you weight the predictions from a few models (can just simply average them for lack of anything better). Using a few models will help prevent any single model from causing a large miss in your forecast. Some of the models worth considering might be: seasonal ...

1

In comments, OP writes: If may help other desperate readers, once the two models were pretty much normalized by the number of parameters and after A LOT of hyper-parameters retuning, the performances started to be comparable. GRU outperforms LSTM but not dramatically, and a consistent difference in training time is visible. I've copied OP's comment as a ...

1

RNNs are directly connected to only the previous time step. However, because the hidden state at that time-step ($t-1$) was influenced by the hidden state at $t-2$, and so on, the RNN can already effectively look back any number of time steps in an indirect manner. You might try to make connections from time $t-k, t-k+1, t-k+2, \ldots, t-1$ to time $t$ but ...

1

In these cases, you usually perform a cross validation, or even better, a k-fold cross validation. The idea is the following: Determine an error threshold Divide your samples into k subsets of the same size (without gaps, potentially overlapping) Repeat until satisfactory performance is reached Select one particular combination of parameters (e.g. ...

1

There are some thousands of variants of RNN cell(kernel) and both LSTM and GRU are for processing the input $x_i$ and the output of the previous state $s_{i-1}$ and producing the output and the current state. Even thought LSTM preceded GRU and GRU contains less computation, LSTM is just on a par with GRU in performance. So, I think stacking LSTM and GRU or ...

1

Does the layrecnet function contain a gate? No. From https://www.mathworks.com/help/nnet/ug/design-layer-recurrent-neural-networks.html?searchHighlight=recurrent%20neural%20network&s_tid=doc_srchtitle: The layrecnet command generalizes the Elman network to have an arbitrary number of layers and to have arbitrary transfer functions in each layer. And ...

1

I don't know if this implementation of a RNN includes a gate, but since you can look at the code you will soon be able to check. The documentation is here.

Only top voted, non community-wiki answers of a minimum length are eligible