I have a three hidden layer neural network. Input layer has 116 nodes(means I have 116 features in every training data set) and output layer has 2 nodes(means I have 2 labels in every training data set).

Strangely enough, every time after training, no matter how low the losses are, only the first column(first node) of the prediction has accurate enough answers. The predictions in the second column is like a random guess, which is not accurate at all.

Like this:

[['0.473684' '0.000000']
['0.947368' '0.052632']
['0.789474' '0.210526']
['0.421053' '0.684211']
['0.052632' '0.789474']
['0.368421' '0.789474']
['0.736842' '0.947368']]

Prediction 1: [0.48385417 0.5978483 ]
Prediction 2: [0.9906172 0.4896417]
Prediction 3: [0.7971876 0.4753807]
Prediction 4: [0.40410563 0.6019936 ]
Prediction 5: [0.0388904 0.55383235]
Prediction 6: [0.34765968 0.60196346]
Prediction 7: [0.6820264 0.59644026]

The upper part are the training labels and the "Prediction" part are the predictions. As you can see the second column is way off while the first is okay accurate( at least comparing to the second column).

I don't really understand why this is happening.

Here I'll provide my NN model:

  def model_fn(features, labels, mode, params):
  """Model function for Estimator."""

  first_hidden_layer = tf.layers.dense(features["x"], 100, activation=tf.nn.leaky_relu)

  second_hidden_layer = tf.layers.dense(first_hidden_layer, 100, activation=tf.nn.leaky_relu)

  third_hidden_layer = tf.layers.dense(second_hidden_layer, 100, activation=tf.nn.leaky_relu)

  # Connect the output layer to third hidden layer (no activation fn)
  output_layer = tf.layers.dense(third_hidden_layer, 2)
  predictions = tf.reshape(output_layer, [-1,2])

  if labels != None:
      labels = tf.reshape(labels, [-1,2])

  var = [v for v in tf.trainable_variables() if "kernel" in v.name]

  # Provide an estimator spec for `ModeKeys.PREDICT`.
  if mode == tf.estimator.ModeKeys.PREDICT:
    return tf.estimator.EstimatorSpec(
        predictions={"par": predictions})

  # Calculate loss using mean squared error
  regularizer = tf.nn.l2_loss(var[0]) + tf.nn.l2_loss(var[1]) + tf.nn.l2_loss(var[2]) + tf.nn.l2_loss(var[3])

  loss = tf.losses.mean_squared_error(labels, predictions)

  optimizer = tf.train.AdamOptimizer(
          learning_rate = params["learning_rate"],
          beta1 = 0.9,
          beta2 = 0.999,
          epsilon = 1e-8)

  train_op = optimizer.minimize( 
      loss=(loss+(beta/2)*regularizer) , global_step=tf.train.get_global_step())

  # Calculate root mean squared error as additional eval metric
  eval_metric_ops = {
      "rmse": tf.metrics.root_mean_squared_error(
          tf.cast(labels, tf.float32), predictions)

  # Provide an estimator spec for `ModeKeys.EVAL` and `ModeKeys.TRAIN` modes.
  return tf.estimator.EstimatorSpec(

The things I did:

  1. I use AdamOptimizer because it works a lot more better than GradientDecentOptimizer.
  2. I scaled all the features and labels to the range of (0,1).
  3. The regularizer here doesn't really matter since I set beta=0. I don't have overfitting problem.
  4. I ran similar settings with only one output and the predictions are pretty accurate. I don't understand why I cannot increase the output nodes to 2 or more.

I think I might have some basic settings wrong or there are something incorrect with my understanding of neural network but I cannot figure them out. Hope someone can point me the right direction. I am open to any discussions. Thanks for enlightening me!


It turns out that the second node has almost no correlation with the features. That's why Tensorflow cannot figure out the weighting between the inputs and output 2 and it is also the reason why the training error cannot go down when it reaches a certain limit.

The take away for this is that ALWAYS verify if your data are reasonable relation-wise between inputs and outputs. Or if one of the outputs cannot be trained ( the predictions are random guesses), maybe re-think the correlation between the data is a good way to tackle the problem.

| cite | improve this answer | |
  • $\begingroup$ Looking at correlations alone in many cases would be not enough. Correlation are about pairwise linear relationships, if your data consisted only of pairwise linear relationships, using simple linear regression would be enough to model it. $\endgroup$ – Tim Jun 5 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.