I'm following Google's deep learning course on Udacity. Assignment 2 is to turn a logistic regression network into a 1-hidden layer neural network with rectified linear units and 1024 hidden nodes using Tensorflow.

To do this I've change this:

logits = tf.matmul(tf_train_dataset, weights) + biases

into this:

logits = tf.matmul(tf.nn.relu(tf.matmul(tf_train_dataset, weights1) +
biases1), weights2) + biases2`

Here is the full code:

def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])


batch_size = 128

graph = tf.Graph()
with graph.as_default():
    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32,
                                      shape=(batch_size, image_size * image_size))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    # Variables.
    weights1 = tf.Variable(
        tf.truncated_normal([image_size * image_size, 1024]))
    biases1 = tf.Variable(tf.zeros([1024]))
    weights2 = tf.Variable(
        tf.truncated_normal([1024, num_labels]))
    biases2 = tf.Variable(tf.zeros([num_labels]))

    # Training computation.
    logits = tf.matmul(tf.nn.relu(tf.matmul(tf_train_dataset, weights1) + biases1), weights2) + biases2
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits))

    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(logits)
    valid_prediction = tf.nn.softmax(
        tf.matmul(
            tf.nn.relu(
                tf.matmul(tf_valid_dataset, weights1)
                + biases1),
            weights2) + biases2)
    test_prediction = tf.nn.softmax(
        tf.matmul(
            tf.nn.relu(
                tf.matmul(tf_test_dataset, weights1)
                + biases1),
            weights2) + biases2)


num_steps = 3001

with tf.Session(graph=graph) as session:
  tf.global_variables_initializer().run()
  print("Initialized")
  for step in range(num_steps):
    # Pick an offset within the training data, which has been randomized.
    # Note: we could use better randomization across epochs.
    offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
    # Generate a minibatch.
    batch_data = train_dataset[offset:(offset + batch_size), :]
    batch_labels = train_labels[offset:(offset + batch_size), :]
    # Prepare a dictionary telling the session where to feed the minibatch.
    # The key of the dictionary is the placeholder node of the graph to be fed,
    # and the value is the numpy array to feed to it.
    feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
    _, l, predictions = session.run(
      [optimizer, loss, train_prediction], feed_dict=feed_dict)
    if (step % 500 == 0):
      print("Minibatch loss at step %d: %f" % (step, l))
      print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
      print("Validation accuracy: %.1f%%" % accuracy(
        valid_prediction.eval(), valid_labels))
  print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

Here is the output I get:

Initialized
Minibatch loss at step 0: 208.975021
Minibatch accuracy: 11.7%
Validation accuracy: 10.0%
Minibatch loss at step 500: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 10.2%
Minibatch loss at step 1000: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 14.6%
Minibatch loss at step 1500: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 10.2%
Minibatch loss at step 2000: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 17.7%
Minibatch loss at step 2500: 2.952326
Minibatch accuracy: 93.8%
Validation accuracy: 26.6%
Minibatch loss at step 3000: 0.000000
Minibatch accuracy: 100.0%
Validation accuracy: 17.5%
Test accuracy: 18.1%

It looks like it's overfitting. It gets close to 100% accuracy on the training data, but only gets around 20% accuracy on the validation and testing data.

I'm confused because I've seen others on Udacity's forum say that they're getting over 90% accuracy using this method. So I think I might be implementing the network incorrectly.

Is this the proper way to implement a 1-hidden layer neural network with rectified linear units? If so, what do you think I'm doing wrong here?

Thank You

  • The method by itself doesn't guarantee any particular accuracy level. It depends on your data and how well separated the classes are. – Michael Chernick May 2 '17 at 2:27
  • @MichaelChernick I'm using the same dataset as the Udacity lesson, the notMNIST dataset. Is there a way that I can check how well separated the classes are without sifting through tens of thousands of images? – Matt D May 2 '17 at 2:30
  • What accuracy did they get that was better than yours? – Michael Chernick May 2 '17 at 2:31
  • I read comments of people getting close to 90% accuracy. Also, what I'm getting now is the same accuracy that I got with a simple linear regression network, which leads me to believe I'm doing something wrong. And the validation accuracy is not increasing with training. – Matt D May 2 '17 at 2:32
  • I am not expert enough about your code to tell if you are doing something wrong. But I don't think you know whether the others are doing it right either. – Michael Chernick May 2 '17 at 2:35

It turns out that I had not shuffled the data before I ran the network. If any other beginners run into a similar issue, make sure that you shuffle the data before you run your experiment so that you get an even mix of classes in each batch.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.