What is the output of a tf.nn.dynamic_rnn()? I am not sure about what I understand from the official documentation, which says:

Returns:
  A pair (outputs, state) where:
outputs: The RNN output Tensor.
If time_major == False (default), this will be a Tensor shaped:
  [batch_size, max_time, cell.output_size].
If time_major == True, this will be a Tensor shaped: [max_time,
  batch_size, cell.output_size].
Note, if cell.output_size is a (possibly nested) tuple of integers or
  TensorShape objects, then outputs will be a tuple having the same
  structure as cell.output_size, containing Tensors having shapes
  corresponding to the shape data in cell.output_size.
state: The final state. If cell.state_size is an int, this will be
  shaped [batch_size, cell.state_size]. If it is a TensorShape, this
  will be shaped [batch_size] + cell.state_size. If it is a (possibly
  nested) tuple of ints or TensorShape, this will be a tuple having the
  corresponding shapes. If cells are LSTMCells state will be a tuple
  containing a LSTMStateTuple for each cell.

Is output[-1] always (in all three cell types i.e. RNN, GRU, LSTM) equal to state (second element of return tuple)? I guess the literature everywhere is too liberal in the use of the term hidden state. Is hidden state in all three cells the score coming out (why it is called hidden is beyond me, it would appear cell state in LSTM should be called the hidden state as it is not exposed)?
 A: Possible copy of https://stackoverflow.com/questions/36817596/get-last-output-of-dynamic-rnn-in-tensorflow/49705930#49705930
Anyway let's go ahead with the answer.
This code snip might help understand what's really being returned by the dynamic_rnn layer 
=> Tuple of (outputs, final_output_state). 
So for an input with max sequence length  of T time steps outputs is of the shape [Batch_size, T, num_inputs] (given time_major=False; default value) and it contains the output state at each timestep h1, h2.....hT. 
And final_output_state is of the shape [Batch_size,num_inputs] and has the final cell state cT and output state hT of each batch sequence. 
But since the dynamic_rnn is being used my guess is your sequence lengths vary for each batch.
    import tensorflow as tf
    import numpy as np
    from tensorflow.contrib import rnn
    tf.reset_default_graph()

    # Create input data
    X = np.random.randn(2, 10, 8)

    # The second example is of length 6 
    X[1,6:] = 0
    X_lengths = [10, 6]

    cell = tf.nn.rnn_cell.LSTMCell(num_units=64, state_is_tuple=True)

    outputs, states  = tf.nn.dynamic_rnn(cell=cell,
                                         dtype=tf.float64,
                                         sequence_length=X_lengths,
                                         inputs=X)

    result = tf.contrib.learn.run_n({"outputs": outputs, "states":states},
                                    n=1,
                                    feed_dict=None)
    assert result[0]["outputs"].shape == (2, 10, 64)
    print result[0]["outputs"].shape
    print result[0]["states"].h.shape
    # the final outputs state and states returned must be equal for each      
    # sequence
    assert(result[0]["outputs"][0][-1]==result[0]["states"].h[0]).all()
    assert(result[0]["outputs"][-1][5]==result[0]["states"].h[-1]).all()
    assert(result[0]["outputs"][-1][-1]==result[0]["states"].h[-1]).all()

The final assertion will fail as the final state for the 2nd sequence is at 6th time step ie. the index 5 and the rest of the outputs from [6:9] are all 0s in the 2nd timestep
