I'm attempting to build an LSTM in Tensorflow to take in a series of amino acids (represented as Bitfields) and output a series of Torsion angles (4 numbers ranging from -1 to 1) for each amino acid in the sequence. As this is ordered data of variable length (up to a maximum of 31), I figured an LSTM was the best kind of network for this.
Most of the examples of LSTMs are based on images (as it's the hot topic of the moment). I'm not classifying here - I'm trying to get as close to the correct angles as possible.
This image is what I think I've built. It kind of makes sense.
Here is the code for the graph:
def create_graph() : graph = tf.Graph() with tf.device('/gpu:0'): with graph.as_default(): # Input has to be [batch_size, max_time, ...] tf_train_dataset = tf.placeholder(tf.int32, [None, FLAGS.max_cdr_length, FLAGS.num_acids],name="train_input") output_size = FLAGS.max_cdr_length * 4 dmask = tf.placeholder(tf.float32, [None, output_size], name="dmask") x = tf.cast(tf_train_dataset, dtype=tf.float32) # Since we are using dropout, we need to have a placeholder, so we dont set # dropout at validation time keep_prob = tf.placeholder(tf.float32, name="keepprob") single_rnn_cell = lstm_cell(FLAGS.lstm_size, keep_prob) # 'outputs' is a tensor of shape [batch_size, max_cdr_length, lstm_size] length = create_length(x) initial_state = single_rnn_cell.zero_state(FLAGS.batch_size, dtype=tf.float32) outputs, state = tf.nn.dynamic_rnn(cell=single_rnn_cell, inputs=x, dtype=tf.float32, sequence_length = length, initial_state = initial_state) # We flatten out the outputs so it just looks like a big batch to our weight matrix # apparently this gives us weights across the entire set of steps output = tf.reshape(outputs, [-1, FLAGS.lstm_size], name="flattened") test = tf.placeholder(tf.float32, [None, output_size], name="train_test") W_i = weight_variable([FLAGS.lstm_size, 4], "weight_intermediate") b_i = bias_variable(,"bias_intermediate") y_i = tf.tanh( ( tf.matmul(output, W_i) + b_i), name="intermediate") # Now reshape it back and run the mask against it y_b = tf.reshape(y_i, [-1, output_size], name="output") y_b = y_b * dmask return graph
Here is the cost function:
def cost(goutput, gtest): # Values of -3.0 are the ones we ignore # This could go wrong as adding 3.0 to -3.0 is not numerically stable mask = tf.sign(tf.add(gtest,3.0)) basic_error = tf.square(gtest-goutput) * mask # reduce mean doesnt work here as we just want the numbers where mask is 1 # We work out the mean ourselves basic_error = tf.reduce_sum(basic_error) basic_error /= tf.reduce_sum(mask) return basic_error
I provide a mask to both the cost function and the final layer of the graph. This mask is 1 for every amino-acid in the sequence, and 0 for where there is not an acid. Basically, I'm padding out the sequence I am interested in with zeroes, up to the maximum length.
My main question is, am I doing the 'right' thing with the weight matrix? I figure that the weights are what is being trained here and they should take into account every acid in the particular sequence. Conceptually, is this what I am doing here?