# Neural network can only follow increasing function

I am trying to program a simple neural network using python. For some reason my code is only working on functions which are increasing. The network I am using has 1 input, 1 output, and 2 layers of 20 neurons. When I train it on the function sin(x) where 0 < x < 1 the output nearly perfectly matches the calculated sin(x). I then tried cos(x) and the output from my network is a constant. I also tried linear functions. These too only work if the function is increasing.

Here is my python code. I am using the cross entropy cost function but I also get the same result with the quadratic cost function.

import numpy as np
import random

def sigma(z):
return 1.0 / (1 + np.exp(z))

def sigma_prime(z):
ez = np.exp(z)
return -ez / (ez + 1)**2

class NeuralNetwork:

def __init__(self, *sizes):
self.w = []
self.b = []
last = sizes
for s in sizes[1:]:
self.w.append(np.full((s, last), 1.0 / (s * last)))
self.b.append(np.zeros(s))
last = s

def proporgate(self, vin):
a = np.array(vin).T
for w, b in zip(self.w, self.b):
a = np.dot(w, a) + b;
a = sigma(a)
return a.flatten()

def randomize(self):
for i in range(len(self.w)):
self.w[i] = np.random.randn(self.w[i].shape, self.w[i].shape)
self.b[i] = np.random.randn(self.b[i].shape)

def train(self, learning_rate, vin, vout):
a = [None] * len(self.w)
z = [None] * len(self.w)

z = np.dot(self.w, np.array(vin).T) + self.b
a = sigma(z)

for i in range(1, len(self.w)):
z[i] = np.dot(self.w[i], a[i - 1]) + self.b[i]
a[i] = sigma(z[i])

d = [None] * len(self.w)
y = np.array(vout).T
#d[-1] = (a[-1] - y) * sigma_prime(z[-1])
d[-1] = ((1 - y) / (1 - a[-1]) - y / a[-1]) * sigma_prime(z[-1])

for i in reversed(range(len(self.w) - 1)):
d[i] = np.dot(self.w[i + 1].T, d[i + 1]) * sigma_prime(z[i])

for i in range(len(self.w)):
self.b[i] -= learning_rate * d[i]
self.w[i] -= learning_rate * np.dot(d[i], a[i].T)

import matplotlib.pyplot as plt

n = NeuralNetwork(1, 20, 20, 1)
n.randomize()
iters = 1000000
for i in range(iters):
x = random.random()
n.train(5 * (1.0 - i / iters), [x], [np.cos(x)])

x = np.linspace(0, 1, 1000)
y = np.zeros_like(x)
for i in range(y.shape):
y[i] = n.proporgate([x[i]])

plt.plot(x, y, x, np.cos(x))
plt.title('cos(x)')
plt.show()


It looks like your network returns the mean of the function, that why you got the line. I checked that mean should be equal approximately 0.84 which is very close to your line. I run your code and check the weights. The weights from the last layer just blow up and become extremely huge.

>>> n.w[-1]
array([[ 53403.18717113,  53402.33451232,  53403.58075363,  53402.85629078,
53403.56506679,  53402.75971252,  53403.78749293,  53402.87328528,
53401.35358122,  53402.36034322,  53402.48372256,  53403.46650581,
53402.50228206,  53401.9487225 ,  53402.93887686,  53402.70398353,
53402.39407981,  53403.3953328 ,  53402.21630115,  53402.72207552]])


You should also get a waring message in the terminal that tell you that something went wrong with you calculations.

nn.py:9: RuntimeWarning: overflow encountered in square


You need to train it with less number of epochs and use smaller step to prevent this behaviour. I used 20,000 epochs and decrease your step by a factor of 5

n.train((1.0 - i / iters), [x], [np.cos(x)])


That's the result that I've gotten. • Thanks. Do you know why the weights tend to blow up? – chasep255 May 4 '16 at 12:50
• Your learning rate is too big and you trained network for too long. You can also use weight decay regularisation to prevent this behaviour. – itdxer May 4 '16 at 12:52
• Do you know if it is bad practice to update the weights after running every training example or should I batch a few training examples and then update the weights and biases? – chasep255 May 4 '16 at 14:15
• batch will give you the average gradient and it will be more accurate than for the one sample. What you are trying to do it's a stochastic gradient descent (SGD). The gradients for the SGD contain a lot of noise. Usualy SGD gives better results, because your make update for each instance separately, but it should defence on the problem. Some algorithms wouldn't work properly in case of mini-batch (or batch with one sample in your case). There are a some pros and cons for both methods. – itdxer May 4 '16 at 14:47