2
$\begingroup$

I am new to tensorflow and am learning the basics at the moment so please bear with me.

My problem concerns strange non-convergent behaviour of neural networks when presented with the supposedly simple task of finding a regression function for a small training set consisting only of m = 100 data points {(x_1, y_1), (x_2, y_2),...,(x_100, y_100)}, where x_i and y_i are real numbers.

I first constructed a function that automatically generates a computational graph corresponding to a classical fully connected feedforward neural network:

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import math

def neural_network_constructor(arch_list = [1,3,3,1], 
                               act_func = tf.nn.sigmoid, 
                               w_initializer = tf.contrib.layers.xavier_initializer(), 
                               b_initializer = tf.zeros_initializer(),
                               loss_function = tf.losses.mean_squared_error,
                               training_method = tf.train.GradientDescentOptimizer(0.5)):

    n_input = arch_list[0]
    n_output = arch_list[-1]

    X = tf.placeholder(dtype = tf.float32, shape = [None, n_input])

    layer = tf.contrib.layers.fully_connected(
            inputs = X,
            num_outputs = arch_list[1],
            activation_fn = act_func,
            weights_initializer = w_initializer,
            biases_initializer = b_initializer)

    for N in arch_list[2:-1]:
        layer = tf.contrib.layers.fully_connected(
                inputs = layer,
                num_outputs = N,
                activation_fn = act_func,
                weights_initializer = w_initializer,
                biases_initializer = b_initializer)

    Phi = tf.contrib.layers.fully_connected(
            inputs = layer,
            num_outputs = n_output,
            activation_fn = tf.identity,
            weights_initializer = w_initializer,
            biases_initializer = b_initializer)


    Y = tf.placeholder(tf.float32, [None, n_output])

    loss = loss_function(Y, Phi)
    train_step = training_method.minimize(loss)

    return [X, Phi, Y, train_step]

With the above default values for the arguments, this function would construct a computational graph corresponding to a neural network with 1 input neuron, 2 hidden layers with 3 neurons each and 1 output neuron. The activation function is per default the sigmoid function. X corresponds to the input tensor, Y to the labels of the training data and Phi to the feedforward output of the neural network. The operation train_step performs one gradient-descent step when executed in the session environment.

So far, so good. If I now test a particular neural network (constructed with this function and the exact default values for the arguments given above) by making it learn a simple regression function for artificial data extracted from a sinewave, strange things happen:

before training

enter image description here

Before training, the network seems to be a flat line. After 100.000 training iterations, it manages to partially learn the function, but only the part which is closer to 0. After this, it becomes flat again. Further training does not decrease the loss function anymore.

This get even stranger, when I take the exact same data set, but shift all x-values by adding 500:

enter image description here enter image description here

Here, the network completely refuses to learn. I cannot understand why this is happening. I have tried changing the architecture of the network and its learning rate, but have observed similar effects: the closer the x-values of the data cloud are to the origin, the easier the network can learn. After a certain distance to the origin, learning stops completely. Changing the activation function from sigmoid to ReLu has only made things worse; here, the network tends to just converge to the average, no matter what position the data cloud is in.

Is there something wrong with my implementation of the neural-network-constructor? Or does this have something do do with initialization values? I have tried to get a deeper understanding of this problem now for quite a while and would greatly appreciate some advice. What could be the cause of this? All thoughts on why this behaviour is occuring are very much welcome!

Thanks, Joker

$\endgroup$
  • $\begingroup$ input data needs to be normalized/standardized in order to properly train. this explains why adding 500 caused everything to stop working. also learning rate of 0.5 is way too high. try 0.01 or 0.001 $\endgroup$ – shimao May 10 '18 at 20:50
  • $\begingroup$ Thank you for your comment. I see. But what is the deeper reason behind this necessity for normalization? If I choose the neural network architecture big enough, it should be able to approximate any given data cloud (no matter where it is positioned) since neural networks are universal approximators, right? $\endgroup$ – Joker123 May 10 '18 at 21:43
  • 1
    $\begingroup$ neural networks are quite difficult to optimize for a variety of reasons, which has mostly been "solved" by clever initialization schemes, more computation power, better variants of gradient descent, etc. often, all of these details are tuned with the assumption that the data is normalized, so that the gradients aren't too large/small and you don't run into numerical problems. Therefore normalizing is necessary. $\endgroup$ – shimao May 10 '18 at 21:48
  • $\begingroup$ @shimao I have written up your comments as an answer. If you would like to take credit for this answer, please feel free to add your own answer & let me know so that I can delete my answer quoting you. $\endgroup$ – Sycorax Jul 5 '18 at 14:45
1
$\begingroup$

User @shimao writes in comments:

input data needs to be normalized/standardized in order to properly train. this explains why adding 500 caused everything to stop working. also learning rate of 0.5 is way too high. try 0.01 or 0.001 neural networks are quite difficult to optimize for a variety of reasons, which has mostly been "solved" by clever initialization schemes, more computation power, better variants of gradient descent, etc. often, all of these details are tuned with the assumption that the data is normalized, so that the gradients aren't too large/small and you don't run into numerical problems. Therefore normalizing is necessary.


I want to clarify that I've this comment as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.

Please review

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.