# Is my weight matrix *learning* from all the steps in my LSTM?

I'm attempting to build an LSTM in Tensorflow to take in a series of amino acids (represented as Bitfields) and output a series of Torsion angles (4 numbers ranging from -1 to 1) for each amino acid in the sequence. As this is ordered data of variable length (up to a maximum of 31), I figured an LSTM was the best kind of network for this.

Most of the examples of LSTMs are based on images (as it's the hot topic of the moment). I'm not classifying here - I'm trying to get as close to the correct angles as possible.

This image is what I think I've built. It kind of makes sense.

Here is the code for the graph:

def create_graph() :
graph = tf.Graph()

with tf.device('/gpu:0'):
with graph.as_default():
# Input has to be [batch_size, max_time, ...]
tf_train_dataset = tf.placeholder(tf.int32, [None, FLAGS.max_cdr_length, FLAGS.num_acids],name="train_input")
output_size = FLAGS.max_cdr_length * 4
x = tf.cast(tf_train_dataset, dtype=tf.float32)

# Since we are using dropout, we need to have a placeholder, so we dont set
# dropout at validation time
keep_prob = tf.placeholder(tf.float32, name="keepprob")

single_rnn_cell = lstm_cell(FLAGS.lstm_size, keep_prob)
# 'outputs' is a tensor of shape [batch_size, max_cdr_length, lstm_size]

length = create_length(x)
initial_state = single_rnn_cell.zero_state(FLAGS.batch_size, dtype=tf.float32)
outputs, state = tf.nn.dynamic_rnn(cell=single_rnn_cell, inputs=x, dtype=tf.float32, sequence_length = length, initial_state = initial_state)

# We flatten out the outputs so it just looks like a big batch to our weight matrix
# apparently this gives us weights across the entire set of steps
output = tf.reshape(outputs, [-1, FLAGS.lstm_size], name="flattened")

test = tf.placeholder(tf.float32, [None, output_size], name="train_test")

W_i = weight_variable([FLAGS.lstm_size, 4], "weight_intermediate")
b_i = bias_variable([4],"bias_intermediate")
y_i = tf.tanh( ( tf.matmul(output, W_i) + b_i), name="intermediate")

# Now reshape it back and run the mask against it
y_b = tf.reshape(y_i, [-1, output_size], name="output")

return graph


Here is the cost function:

def cost(goutput, gtest):
# Values of -3.0 are the ones we ignore
# This could go wrong as adding 3.0 to -3.0 is not numerically stable

# reduce mean doesnt work here as we just want the numbers where mask is 1
# We work out the mean ourselves
basic_error = tf.reduce_sum(basic_error)
return basic_error


I provide a mask to both the cost function and the final layer of the graph. This mask is 1 for every amino-acid in the sequence, and 0 for where there is not an acid. Basically, I'm padding out the sequence I am interested in with zeroes, up to the maximum length.

My main question is, am I doing the 'right' thing with the weight matrix? I figure that the weights are what is being trained here and they should take into account every acid in the particular sequence. Conceptually, is this what I am doing here?

Cheers

Ben

• No idea but +1 for your drawing. – David Kozak Nov 22 '17 at 18:10
• Still havent found a good diagram drawing tool outside of omnigraffle and that is still slower :) – Oni Nov 22 '17 at 18:50
• I have many questions about this design but in an attempt to be more specific I'm trying to focus on one bit. Figuring out the connection between tensorflow and the architecture is perhaps the hardest bit. – Oni Nov 22 '17 at 18:51
• it's thanksgiving in the US, you're not going to get any answers until Monday :) – Aksakal Nov 22 '17 at 21:44
• So running this, it actually performs worse than I thought. Am I right in thinking that the the earlier the step, the more represented it is in the final weight matrix? I mean, the 1st step happens, then the 2nd step on top of the first step happens - thus the 1st step is represented again, at least partially. So perhaps this method is not really that great. – Oni Dec 3 '17 at 15:53