# Deep Learning for sequences

I want to use deep learning techniques to perform better inference tasks than Hidden Markov Models (which is a shallow model)? I was wondering what is the state-of-the art deep learning model to replace Hidden Markov Models (HMM)? The set-up is semi-supervised. The training data X(t),Y(t) is a time series, with significant temporal correlations. Also, there is a huge amount of unlabelled data, i.e., simply X(t) and no Y(t). After reading many papers, I narrowed down on the following model -> Conditionally Restricted Boltzmann Machines (Ilya Sustkever MS thesis) and use Deep Belief Networks for unsupervised pretraining (or use variational autoencoders for pretraining). I am very new to the field, and was wondering if these techniques are outdated.

• the state-of-the-art currently for this task is undoubtedly Recurrent Neural Networks (LSTMs and GRUs) – Antoine May 10 '17 at 14:31
• @antoine the "recurrent" part in RNN is basically a compact way to store past history by allowing for "cycles" in the ANN graph, and I am well aware of it. Though I know that training the RNNs is tricky, there has been recent progress in training RNNs (Ilya Sustkever's Phd thesis), I would like to start with something more basic. – rahuls88 May 10 '17 at 14:51
• Unsupervised pre-training is no longer state of the art -- advances in regularization methods (dropout, max norm) and transfer functions (ReLU and variants) make that unnecessary. Antoine is correct that RNNs are the state of the art in deep learning for sequences -- to the point that RNNs and deep learning for sequences are, effectively, synonyms. – Sycorax says Reinstate Monica May 10 '17 at 14:55
• @rahuls88 RNNs indeed have been around for a long time, but the LSTMs and GRUs architectures are much more recent and considered cutting edge, as far as I know. And within the context of your question (how to improve on HMMs), LSTMs/GRUs implement tricks to preserve their memory over time while taking into account the new observations, so they can capture dependence between 2 elements in the series for a much longer time than HMMs (which make the Markov assumption). Check out the RNN tutorial by Denny Britz (from Google Brain), it is really good – Antoine May 10 '17 at 15:07
• @antoine looks good....will try RNNs also – rahuls88 May 10 '17 at 17:35

In addition to the various RNNs mentioned in the comments, another type of layer that can be used when both the input and the output is a sequence is the following CRF-style layer (gave state-of-the-art results for named-entity recognition as in the paper mentioned below, as well as sequential short-text classification). I have only used it for fully supervised tasks though.

https://arxiv.org/abs/1606.03475 (De-identification of Patient Notes with Recurrent Neural Networks) uses a neural network with a "label sequence optimization layer" as the top layer to do some sequence tagging, which could be seen as a "deep learning" equivalent to CRF.

See Section 2.2.4 Label sequence optimization layer:

The label sequence optimization layer takes the sequence of probability vectors $\mathbf{a}_{1:n}$ from the label prediction layer as input, and outputs a sequence of labels $y_{1:n}$, where $y_{i}$ is the label assigned to the token $x_{i}$.

The simplest strategy to select the label $y_{i}$ would be to choose the label that has the highest probability in $\mathbf{a}_{i}$, i.e. $y_{i}=\text{argmax}_{k}{\mathbf{a}_{i}[k]}$. However, this greedy approach fails to take into account the dependencies between subsequent labels. For example, it may be more likely to have a token with the PHI type STATE followed by a token with the PHI type ZIP than any other PHI type. Even though the label prediction layer has the capacity to capture such dependencies to a certain degree, it may be preferable to allow the model to directly learn these dependencies in the last layer of the model.

One way to model such dependencies is to incorporate a matrix $T$ that contains the transition probabilities between two subsequent labels. $T[i,j]$ is the probability that a token with label $i$ is followed by a token with the label $j$. The score of a label sequence $y_{1:n}$ is defined as the sum of the probabilities of individual labels and the transition probabilities: $$s(y_{1:n}) = { \sum_{i=1}^{n} > \mathbf{a}_{i}[y_{i}]+ \sum_{i=2}^{n} T [y_{i-1},y_{i}} ].$$ These scores can be turned into probabilities of the label sequences by taking a softmax function over all possible label sequences. During the training phase, the objective is to maximize the log probability of the gold label sequence. In the testing phase, given an input sequence of tokens, the corresponding sequence of predicted labels is chosen as the one that maximizes the score.

The network:

• @Frank thanks for the reference...looks good, I will try it out – rahuls88 May 10 '17 at 15:43