Conditional Independence in Recurrent Neural Work For the simplest case of recurrent neural network, is the hidden state at time t+1 conditional independent from the hidden state at time t-1 given hidden state at time t?
 A: The hidden states in a typical recurrent neural network (RNN) are not random variables, since they are deterministically computed, unlike the hidden states of an HMM, which are random variables. Since the concept of conditional independence belongs to probability theory, it doesn't make sense to talk about conditional independence between hidden states in an RNN.
But it is correct that to compute $h_{t +1}$ (hidden state at time t+1), if we know $h_{t}$ we don't need $h_{t -1}$.
Note that one can add random variable in a recurrent neural network, e.g. : 
Chung, Junyoung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. "A recurrent latent variable model for sequential data." In Advances in neural information processing systems, pp. 2980-2988. 2015. http://papers.nips.cc/paper/5653-joint-training-of-a-convolutional-network-and-a-graphical-model-for-human-pose-estimation.pdf

We introduce a recurrent version of the VAE for the purpose of modelling sequences.
  Drawing inspiration from simpler dynamic Bayesian networks (DBNs) such as HMMs and Kalman
  filters, the proposed variational recurrent neural network (VRNN) explicitly models the dependencies
  between latent random variables across subsequent timesteps. However, unlike these simpler
  DBN models, the VRNN retains the flexibility to model highly non-linear dynamics

