# 2-output node Neural network. Only the first output node can predict accurate enough results

I have a three hidden layer neural network. Input layer has 116 nodes(means I have 116 features in every training data set) and output layer has 2 nodes(means I have 2 labels in every training data set).

Strangely enough, every time after training, no matter how low the losses are, only the first column(first node) of the prediction has accurate enough answers. The predictions in the second column is like a random guess, which is not accurate at all.

Like this:

[['0.473684' '0.000000']
['0.947368' '0.052632']
['0.789474' '0.210526']
['0.421053' '0.684211']
['0.052632' '0.789474']
['0.368421' '0.789474']
['0.736842' '0.947368']]

Prediction 1: [0.48385417 0.5978483 ]
Prediction 2: [0.9906172 0.4896417]
Prediction 3: [0.7971876 0.4753807]
Prediction 4: [0.40410563 0.6019936 ]
Prediction 5: [0.0388904 0.55383235]
Prediction 6: [0.34765968 0.60196346]
Prediction 7: [0.6820264 0.59644026]

The upper part are the training labels and the "Prediction" part are the predictions. As you can see the second column is way off while the first is okay accurate( at least comparing to the second column).

I don't really understand why this is happening.

Here I'll provide my NN model:

  def model_fn(features, labels, mode, params):
"""Model function for Estimator."""

first_hidden_layer = tf.layers.dense(features["x"], 100, activation=tf.nn.leaky_relu)

second_hidden_layer = tf.layers.dense(first_hidden_layer, 100, activation=tf.nn.leaky_relu)

third_hidden_layer = tf.layers.dense(second_hidden_layer, 100, activation=tf.nn.leaky_relu)

# Connect the output layer to third hidden layer (no activation fn)
output_layer = tf.layers.dense(third_hidden_layer, 2)
predictions = tf.reshape(output_layer, [-1,2])

if labels != None:
labels = tf.reshape(labels, [-1,2])

var = [v for v in tf.trainable_variables() if "kernel" in v.name]

# Provide an estimator spec for ModeKeys.PREDICT.
if mode == tf.estimator.ModeKeys.PREDICT:
return tf.estimator.EstimatorSpec(
mode=mode,
predictions={"par": predictions})

# Calculate loss using mean squared error
regularizer = tf.nn.l2_loss(var[0]) + tf.nn.l2_loss(var[1]) + tf.nn.l2_loss(var[2]) + tf.nn.l2_loss(var[3])

loss = tf.losses.mean_squared_error(labels, predictions)

learning_rate = params["learning_rate"],
beta1 = 0.9,
beta2 = 0.999,
epsilon = 1e-8)

train_op = optimizer.minimize(
loss=(loss+(beta/2)*regularizer) , global_step=tf.train.get_global_step())

# Calculate root mean squared error as additional eval metric
eval_metric_ops = {
"rmse": tf.metrics.root_mean_squared_error(
tf.cast(labels, tf.float32), predictions)
}

# Provide an estimator spec for ModeKeys.EVAL and ModeKeys.TRAIN modes.
return tf.estimator.EstimatorSpec(
mode=mode,
loss=loss,
train_op=train_op,
eval_metric_ops=eval_metric_ops)


The things I did:

1. I use AdamOptimizer because it works a lot more better than GradientDecentOptimizer.
2. I scaled all the features and labels to the range of (0,1).
3. The regularizer here doesn't really matter since I set beta=0. I don't have overfitting problem.
4. I ran similar settings with only one output and the predictions are pretty accurate. I don't understand why I cannot increase the output nodes to 2 or more.

I think I might have some basic settings wrong or there are something incorrect with my understanding of neural network but I cannot figure them out. Hope someone can point me the right direction. I am open to any discussions. Thanks for enlightening me!