# Tag Info

42

Background We first have to go over some concepts from the theory of computation. An algorithm is a procedure for calculating a function. Given the input, the algorithm must produce the correct output in a finite number of steps and then terminate. To say that a function is computable means that there exists an algorithm for calculating it. Among the ...

33

All RNNs have feedback loops in the recurrent layer. This lets them maintain information in 'memory' over time. But, it can be difficult to train standard RNNs to solve problems that require learning long-term temporal dependencies. This is because the gradient of the loss function decays exponentially with time (called the vanishing gradient problem). LSTM ...

27

You have a dataset containing: images I1, I2, ... ground truth texts T1, T2, ... for the images I1, I2, ... So your dataset could look something like that: A Neural Network (NN) outputs a score for each possible horizontal position (often called time-step t in the literature) of the image. This looks something like this for a image with width 2 (t0, t1) ...

24

I'll assume we're talking about recurrent neural nets (RNNs) that produce an output at every time step (if output is only available at the end of the sequence, it only makes sense to run backprop at the end). RNNs in this setting are often trained using truncated backpropagation through time (BPTT), operating sequentially on 'chunks' of a sequence. The ...

20

Attention is a method for aggregating a set of vectors $v_i$ into just one vector, often via a lookup vector $u$. Usually, $v_i$ is either the inputs to the model or the hidden states of previous time-steps, or the hidden states one level down (in the case of stacked LSTMs). The result is often called the context vector $c$, since it contains the context ...

18

I think that you are referring to vertically stacked LSTM layers (assuming the horizontal axes is the time axis. In that case the main reason for stacking LSTM is to allow for greater model complexity. In case of a simple feedforward net we stack layers to create a hierarchical feature representation of the input data to then use for some machine learning ...

17

The accepted answer focuses on the practical side of the question: it would require a lot of resources, if there parameters are not shared. However, the decision to share parameters in an RNN has been made when any serious computation was a problem (1980s according to wiki), so I believe it wasn't the main argument (though still valid). There are pure ...

17

The terminology is unfortunately inconsistent. num_units in TensorFlow is the number of hidden states, i.e. the dimension of $h_t$ in the equations you gave. Also, from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard9/tf.nn.rnn_cell.RNNCell.md : The definition of cell in this package ...

16

To fully answer this question, it would require a lot of pages here. Don't forget, stackexchange is not a textbook from which people read for you. Multi-layered perceptron (MLP): are the neural networks that (probably) started everything. They are strictly feed-forward (one directional), i.e. a node from one layer can only have connections to a node of the ...

16

I have done some projects on text classification and relation extraction using CNN and RNN (specifically, LSTM and GRU): CNNs tend to be much faster (~5 times faster) than RNN. It's hard to draw fair comparisons: CNN and RNN have different hyperparameters (filter dimension, number of filters, hidden state dimension, etc.) there exist many sort of RNNs the ...

16

I found this just below the [samples, time_steps, features] you are concerned with. X = numpy.reshape(dataX, (len(dataX), seq_length, 1)) Samples - This is the len(dataX), or the amount of data points you have. Time steps - This is equivalent to the amount of time steps you run your recurrent neural network. If you want your network to have memory of 60 ...

13

One can use RNN to map multiple input to a single input (label), as this give figure (source) illustrates: Each rectangle is a vector and arrows represent functions (e.g. matrix multiply). Input vectors are in red, output vectors are in blue and green vectors hold the RNN's state (more on this soon). From left to right: (1) Vanilla mode of processing ...

12

The 'shared weights' perspective comes from thinking about RNNs as feedforward networks unrolled across time. If the weights were different at each moment in time, this would just be a feedforward network. But, I suppose another way to think about it would be as an RNN whose weights are a time-varying function (and that could let you keep the ability to ...

12

I think there's some confusion here. The reason you have vanishing gradients in neural networks (with say, softmax) is wholly different from RNNs. With neural networks, you get vanishing gradients because most initial conditions make your outputs end up on either the far left or far right of your softmax layer, giving it a vanishingly small gradient. In ...

12

A very short answer: LSTM decouples cell state (typically denoted by c) and hidden layer/output (typically denoted by h), and only do additive updates to c, which makes memories in c more stable. Thus the gradient flows through c is kept and hard to vanish (therefore the overall gradient is hard to vanish). However, other paths may cause gradient explosion. ...

10

The Bengio et al article "On the difficulty of training recurrent neural networks" gives a hint as to why L2 regularization might kill RNN performance. Essentially, L1/L2 regularizing the RNN cells also compromises the cells' ability to learn and retain information through time. Using an L1 or L2 penalty on the recurrent weights can help with exploding ...

10

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights). Skip connections form conceptually a '...

9

From my understanding, the BPC is just the average cross-entropy (used with log base 2). In the case of Alex Graves' papers, the aim of the model is to approximate the probability distribution of the next character given past characters. At each time step $t$, let's call this (approximate) distribution $\hat{P}_t$ and let $P_t$ be the true distribution. ...

9

According to Rahul Dey and Fathi M. Salem, "Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks": ... the total number of parameters in the GRU RNN equals $3 \times (n^2 + nm + n)$. where $m$ is the input dimension and $n$ is the output dimension. This is due to the fact that there are three sets of operations requiring weight matrices of these ...

9

Yes, cell output equals to the hidden state. In case of LSTM, it's the short-term part of the tuple (second element of LSTMStateTuple), as can be seen in this picture: But for tf.nn.dynamic_rnn, the returned state may be different when the sequence is shorter (sequence_length argument). Take a look at this example: n_steps = 2 n_inputs = 3 n_neurons = 5 X ...

8

UPDATED Your example is very interesting. On one hand it is constructed in such a way that you really need only one parameter and its value is 1: $$y_t=\beta+w y_{t-1}\\\beta=0\\w=1$$ Your training data set is small (96 observations), but with three layer network you have quite a few parameters. It's very easy to overfit. The most interesting part is your ...

8

The problem you pose here is called named entity recognition (NER), or named entity extraction. There are multiple technologies (not necessary neural networks) that can be used for this problem, and some of them are quite mature. See e.g. this repo for an easy-to-plug-in solution, or try to apply the ne_chunk_sents function from the NLTK module in Python.

7

Because RNN is trained by backpropagation through time, and therefore unfolded into feed forward net with multiple layers. When gradient is passed back through many time steps, it tends to grow or vanish, same way as it happens in deep feedforward nets

7

Embeddings are dense vector representations of the characters. The rationale behind using it is to convert an arbitrary discrete id, to a continuous representation. The main advantage is that back-propagation is possible over continuous representations while it is not over discrete representations. A second advantage is that the vector representation might ...

7

Summary Hidden Markov Models (HMMs) are much simpler than Recurrent Neural Networks (RNNs), and rely on strong assumptions which may not always be true. If the assumptions are true then you may see better performance from an HMM since it is less finicky to get working. An RNN may perform better if you have a very large dataset, since the extra complexity ...

7

According to Ioffe and Szegedy (2015), batch normalization is employed to stabilize the inputs to nonlinear activation functions. "Batch Normalization seeks a stable distribution of activation values throughout training, and normalizes the inputs of a nonlinearity since that is where matching the moments is more likely to stabilize the distribution" So ...

7

Leave one out cross validation for LSTM, or any other time-series model, doesn't really make much sense because it would introduce missing values in the series and leaking information from the future. Time series models learn from historical values, to predict future. In leave one out cross-validation, you remove observations from the series, including the ...

6

You can maybe get some inspiration from the ideas presented in this article which proposes a representation of the time series such that it deals with asynchronous sampling: you encode what is the source (the id of the time series) and the duration (time to the last value considering all time series) of the current value, and you end up with a single time ...

6

From {1}: While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that deep RNNs work better than shallower ones on some tasks. In particular, Sutskever et al (2014) report that a 4-layers deep architecture was crucial in achieving good machine-translation performance in an ...

6

There are different approaches Recursive strategy one many-to-one model prediction(t+1) = model(obs(t-1), obs(t-2), ..., obs(t-n)) prediction(t+2) = model(prediction(t+1), obs(t-1), ..., obs(t-n)) Direct strategy multiple many-to-one models prediction(t+1) = model1(obs(t-1), obs(t-2), ..., obs(t-n)) prediction(t+2) = model2(obs(t-2), obs(t-3), ..., ...

Only top voted, non community-wiki answers of a minimum length are eligible