# Neural network prediction is not tending towards the expected value

I am trying to build my understanding of neural networks by implementing a very simple neural network that has just one input and one output in Python.

Here's the code:

import numpy as np

class NeuralNetwork(object):
def __init__(self, update_coefficient):
self.weight = np.random.randn()
self.predicted_output = 0
self.update_coefficient = update_coefficient

def sigmoid(self, val):
res = 1.0 / (1.0 + np.exp(-val))
return res

def sigmoid_derivative(self, val):
res = val * (1 - val)
return res

def get_cost(self, input, expected_output):
self.predicted_output = self.sigmoid(input * self.weight)
res = 0.5 * ((expected_output - self.predicted_output) ** 2)
return res

def get_cost_derivative_wrt_weight(self, input, expected_output):
cost_deriv = ((expected_output - self.predicted_output) *
self.sigmoid_derivative(self.predicted_output) *
input)
return cost_deriv

def update_weight(self, input, expected_output):
self.weight += (self.update_coefficient *
self.get_cost_derivative_wrt_weight(input, expected_output))

nn = NeuralNetwork(1.0)

input = 3; predicted = 2

for var in range(1000):
nn.get_cost(input, predicted)
nn.update_weight(input, predicted)


What is puzzling me is that the predicted value from the network is converging, but it is converging to 1, rather than 2.

What am I doing wrong here?

• Hi Lukasz Grad, thanks so much - I found that when I removed the sigmoid parts of the code, my network worked like a dream. Can you post your comment as an answer so that I can mark it as accepted? Also, is there a point to sigmoid functions in neural networks if you don't need the network to classify a set of inputs as either 0 or 1? – Tola Odejayi Apr 16 '17 at 22:03

Simple, sigmoid is a function $ℜ→(0,1)$, so $1$ is as far as it can get.
Typically, we use sigmoids (or other non-linear functions) in hidden layers of network. Acitvation function in output layer is dependent on the domain of values we want to predict. It can simply be identity if the output is in whole $\Re$ set or sigmoid if we need e.g. probability estimates in $[0,1]$ range. Hope it helps