I'm trying to figure out how to decrease the error in my LSTM. It's an odd use-case because rather than classifying, we are taking in short lists (up to 32 elements long) and outputting a series of real numbers, ranging from -1 to 1 - representing angles. Essentially, we want to reconstruct short protein loops from amino acid inputs.

In the past we had redundant data in our datasets, so the accuracy reported was incorrect. Since removing the redundant data our validation accuracy has gotten much worse, which suggests our network had learned to memorise the most frequent examples.

Our dataset is 10,000 items, split 70/20/10 between train, validation and test. We use a bi-directional, LSTM as follows:

x = tf.cast(tf_train_dataset, dtype=tf.float32)
output_size = FLAGS.max_cdr_length * 4
dmask = tf.placeholder(tf.float32, [None, output_size], name="dmask")
keep_prob = tf.placeholder(tf.float32, name="keepprob")
sizes = [FLAGS.lstm_size,int(math.floor(FLAGS.lstm_size/2)),int(math.floor(FLAGS.lstm_size/     4))]
single_rnn_cell_fw = tf.contrib.rnn.MultiRNNCell( [lstm_cell(sizes[i], keep_prob, "cell_fw" + str(i)) for i in range(len(sizes))])
single_rnn_cell_bw = tf.contrib.rnn.MultiRNNCell( [lstm_cell(sizes[i], keep_prob, "cell_bw" + str(i)) for i in range(len(sizes))])

length = create_length(x)
initial_state = single_rnn_cell_fw.zero_state(FLAGS.batch_size, dtype=tf.float32)
initial_state = single_rnn_cell_bw.zero_state(FLAGS.batch_size, dtype=tf.float32)

outputs, states = tf.nn.bidirectional_dynamic_rnn(cell_fw=single_rnn_cell_fw, cell_bw=single_rnn_cell_bw, inputs=x, dtype=tf.float32, sequence_length = length)
output_fw, output_bw = outputs
states_fw, states_bw = states
output_fw = last_relevant(FLAGS, output_fw, length, "last_fw")
output_bw = last_relevant(FLAGS, output_bw, length, "last_bw")
output = tf.concat((output_fw, output_bw), axis=1, name='bidirectional_concat_outputs')

test = tf.placeholder(tf.float32, [None, output_size], name="train_test")
W_o = weight_variable([sizes[-1]*2, output_size], "weight_output")
b_o = bias_variable([output_size],"bias_output")
y_conv = tf.tanh( ( tf.matmul(output, W_o)) * dmask, name="output")

Essentially, we use 3 layers of LSTM, with 256, 128 and 64 units each. We take the last step of both the Forward and Backward passes and concatenate them together. These feed into a final, fully connected layer that presents the data in the way we need it. We use a mask to set these steps we don't need to zero.

Our cost function uses a mask again, and takes the mean of the squared difference. We build the mask from the test data. Values to ignore are set to -3.0.

def cost(goutput, gtest, gweights, FLAGS):
    mask = tf.sign(tf.add(gtest,3.0))
    basic_error = tf.square(gtest-goutput) * mask
    basic_error = tf.reduce_sum(basic_error)
    basic_error /= tf.reduce_sum(mask)
    return basic_error

To train the net I've used a variety of optimizers. The lowest scores have been obtained with the AdamOptimizer. The others, such as Adagrad, Adadelta, RMSProp tend to flatline around 0.3/0.4 error which is not particularly great.

Our learning rate is 0.004, batch size of 200. We use a 0.5 probability dropout layer.

I've tried adding more layers, changing learning rates, batch sizes, even the representation of the data. I've attempted batch regularisation, L1 and L2 weight regularisation (though perhaps incorrectly) and I've even considered switching to a convnet approach instead.

Nothing seems to make any difference. What has seemed to work is changing the optimizer. Adam seems noisier as it improves, but it does get closer than the other optimizers.

enter image description here

We need to get down to a value much closer to 0.05 or 0.01. Sometimes the training error touches 0.09 but the validation doesn't follow. I've run this network for about 500 epochs so far (about 8 hours) and it tends to settle around 0.2 validation error.

I'm not quite sure what to attempt next. Decayed learning rate might help but I suspect there is something more fundamental I need to do. It could be something as simple as a bug in the code - I need to double check the masking,

  • 1
    $\begingroup$ Do you have any indicator that it is even possible to get lower error with the dataset you have? $\endgroup$ Commented May 2, 2019 at 9:19

1 Answer 1


Ok So I know very little about ANNs. This being said I read recently that a successful strategy for getting lower errors is sometime to use a large and oscillating learning rate (ref). Sounds easy enough to implement, so why not try.

Also, might it not be so that you have actually reached the limit of information that can be extracted from the data?

  • $\begingroup$ Certainly worth a shot, the oscillating learning rate and easy to implement. Thanks. Why I suspect I've gone wrong is that I've changed some quite fundamental things and seen little difference. One in particular is going from a sparse, bitfield approach to input data, to a dense, 5D vector representation and seen little change. The latter representation provides more information so I'd expect to see some kind of change. $\endgroup$
    – Oni
    Commented May 22, 2018 at 14:13
  • $\begingroup$ It does sound suspect. Calls for a very thorough debugging :) $\endgroup$ Commented May 22, 2018 at 18:24
  • $\begingroup$ How's it going? It occurred to me that in my course of ANN I was taught that a good first sanity check of the architecture is to try to overfit the data. If it does, it means the implementation is correct. Don't know if/how this applies to LSTMs $\endgroup$ Commented May 31, 2018 at 10:25
  • $\begingroup$ Thats an interesting idea. I don't believe I've managed to overfit yet. I've not seen the validation error increase as the training increases. $\endgroup$
    – Oni
    Commented May 31, 2018 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.