Deep Learning for sequences I want to use deep learning techniques to perform better inference tasks than Hidden Markov Models (which is a shallow model)? I was wondering what is the state-of-the art deep learning model to replace Hidden Markov Models (HMM)? The set-up is semi-supervised. The training data X(t),Y(t) is a time series, with significant temporal correlations. Also, there is a huge amount of unlabelled data, i.e., simply X(t) and no Y(t). After reading many papers, I narrowed down on the following model -> Conditionally Restricted Boltzmann Machines (Ilya Sustkever MS thesis) and use Deep Belief Networks for unsupervised pretraining (or use variational autoencoders for pretraining). I am very new to the field, and was wondering if these techniques are outdated.
 A: In addition to the various RNNs mentioned in the comments, another type of layer that can be used when both the input and the output is a sequence is the following CRF-style layer (gave state-of-the-art results for named-entity recognition as in the paper mentioned below, as well as sequential short-text classification). I have only used it for fully supervised tasks though.
https://arxiv.org/abs/1606.03475 (De-identification of Patient Notes with Recurrent Neural Networks) uses a neural network with a "label sequence optimization layer" as the top layer to do some sequence tagging, which could be seen as a "deep learning" equivalent to CRF.
See Section 2.2.4 Label sequence optimization layer:

The label sequence optimization layer takes the sequence of
  probability vectors $\mathbf{a}_{1:n}$ from the label prediction layer
  as input, and outputs a sequence of labels $y_{1:n}$, where $y_{i}$ is
  the label assigned to the token $x_{i}$.
The simplest strategy to select the label $y_{i}$ would be to choose
  the label that has the highest probability in $\mathbf{a}_{i}$, i.e.
  $y_{i}=\text{argmax}_{k}{\mathbf{a}_{i}[k]}$. However, this greedy approach
  fails to take into account the dependencies between subsequent labels.
  For example, it may be more likely to have a token with the PHI type
  STATE followed by a token with the PHI type ZIP than any other PHI
  type. Even though the label prediction layer has the capacity to
  capture such dependencies to a certain degree, it may be preferable to
  allow the model to directly learn these dependencies in the last layer
  of the model.
One way to model such dependencies is to incorporate a matrix $T$ that
  contains the transition probabilities between two subsequent labels.
  $T[i,j]$ is the probability that a token with label $i$ is followed by
  a token with the label $j$. The score of a label sequence $y_{1:n}$ is
  defined as the sum of the probabilities of individual labels and the
  transition probabilities: $$ s(y_{1:n}) = { \sum_{i=1}^{n}
> \mathbf{a}_{i}[y_{i}]+  \sum_{i=2}^{n} T [y_{i-1},y_{i}} ]. $$ These
  scores can be turned into probabilities of the label sequences by
  taking a softmax function over all possible label sequences.  During
  the training phase, the objective is to maximize the log probability
  of the gold label sequence. In the testing phase, given an input
  sequence of tokens, the corresponding sequence of predicted labels is
  chosen as the one that maximizes the score.

The network:

Code: https://github.com/Franck-Dernoncourt/NeuroNER
