Scale back the data after a neural network is trained to approximate a function

I often heard people saying that "it is important to feed in a neural network normalized data in order for it to accurately approximate a function". So I did an experiment to see how unnormalized data affects training (I will attach the code at the very end) In my experiment, I wanted to train a neural network to approximate $$f(x_1,...,x_{10}) = \sum\limits_{i=1}^{10}x_i + (\sum\limits_{i=1}^{10}x_i)^2 + (\sum\limits_{i=1}^{10}x_i)^3$$

First, I generated a training set and testing set on the interval $[-1,1]$ (meaning all $x_i$ are in this interval). I used standard square difference as the cost function, and I used stochastic gradient descent optimizer with batch_size = 100.

The performance of the neural network seems to be fine during training process,i.e, the error during training process went steadily down to almost 0.

Second, I generated a training set and testing set on the interval $[-10, 10]$. Then, I saw the outrageous performance of the neural network on this training set. Simply speaking, it is untrainable. Right after a few iterations, the error becomes Nan, the entries in weight matrices become Nan, everything becomes Nan.

I know the reason, it is because during the gradient descent algorithm the gradient vector is "too long". This justifies to some extend that we should normalize the data before training.

However, my question is how do we actually use the trained neural network to predict the value of the function if the input is beyond $[-1, 1]$?

import tensorflow as tf
import numpy as np

input_vector_length = int(10)
output_vector_length = int(1)
train_data_size = int(50000)
test_data_size = int(10000)
train_input_domain = [-1, 1]
test_input_domain = [-1, 1]
iterations = 50000
batch_size = 100
sess = tf.Session()

x = tf.placeholder(tf.float32, shape=[None, input_vector_length], name="x")
y = tf.placeholder(tf.float32, shape =[None, output_vector_length], name="y")

function = tf.reduce_sum(x, 1) + tf.pow(tf.reduce_sum(x,1), 2) + tf.pow(tf.reduce_sum(x,1), 3)

#make train data input
train_input = (train_input_domain[1]-    train_input_domain[0])*np.random.rand(train_data_size, input_vector_length) + train_input_domain[0]

#make train data label
train_label = sess.run(function, feed_dict = {x : train_input})
train_label = train_label.reshape(train_data_size, output_vector_length)

#make test data input
test_input = (test_input_domain[1]-test_input_domain[0])*np.random.rand(test_data_size, input_vector_length) + test_input_domain[0]

#make test data label
test_label = sess.run(function, feed_dict = {x : test_input})
test_label = test_label.reshape(test_data_size, output_vector_length)

def weight_variables(shape, name):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variables(shape, name):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def take_this_batch(data, batch_index=[]):
A = []
for i in range(len(batch_index)):
A.append(data[i])
return A

W_0 = weight_variables(shape=[input_vector_length, 10], name="W_0")
B_0 = bias_variables(shape=[10], name="W_0")
y_1 = tf.sigmoid(tf.matmul(x, W_0) + B_0)

W_output = weight_variables(shape=[10, output_vector_length], name="W_output")
B_output = bias_variables(shape=[output_vector_length], name="B_output")
y_output = tf.matmul(y_1, W_output) + B_output

error = tf.reduce_mean(tf.square(y - y_output))
cost = error

sess.run(tf.global_variables_initializer())

with sess.as_default():
for step in range(iterations):
batch_index = np.random.randint(low=0, high=train_data_size, size=batch_size)
batch_input = take_this_batch(train_input, batch_index)
batch_label = take_this_batch(train_label, batch_index)
train_step.run(feed_dict = {x : batch_input, y:batch_label})
if step % 1000 == 0:
current_error = error.eval(feed_dict = {x:batch_input, y:batch_label})
print("step %d, Current error is %f" % (step,current_error))

print(error.eval(feed_dict={x:test_input, y:test_label}))
e


Usually we normalise the training data in order to fit it in the interval $[-1, 1]$. Then you will apply the same transformation to your validation and testing data (this means you use min and max of the training population). So the data set building procedure for your experiment should be the following:

1. Random sample your $x_i$, let's say within the interval $[-10, 10]$;
2. Compute the whole data set (apply your $f(\boldsymbol{x})$);
3. Split the data set in 60% - 20% - 20% (train, validation, test) chunks, if you care about cross validate your model, or 70% - 30% (train, test), if you don't.
4. Normalise the train data (get train_min and train_max).
5. Apply same transformation to the (validation and) test data.
6. Train on the training data.
7. (Validate and) test your model.

This should be pretty much it.

• I actually have tried to normalize the data before training. The good thing is the neural network became trainable, i.e. all Nan dissappeared. The bad thing is it still has large error when I used the trained neural net to evaluate a number outside [-1, 1]. I guess neural net can only perform well locally. If the train data set is within [-10, 10], we should split it into 10 small pieces, and bring each of them back to [-1, 1] by a linear transformation. Commented Feb 4, 2017 at 14:58
• No. As I said, your network is as good as your training data. It doesn't care how you generate your data (sampling from whatever $[a, b]$ interval for $x_i$). As long as your training data is well representing the whole population, it'll do fine. Still, you have to perform regularisation on the training (and apply the same transformation on validation and test) data set. So, your input data $f(\boldsymbol{x})$ will be always within $[-1, 1]$. Commented Feb 4, 2017 at 15:14