From what I understand, the hidden states of RNNs are equivalent to the deterministic probability distribution over hidden states in for example a Hidden Markov Model.

Thus, just as probabilistic models such as Markov chains or HMM, have state transition probabilities, which forms the transition matrix, does there exist a similar state transition function in recurrent neural networks?