I am trying to build my understanding of neural networks by implementing a very simple neural network that has just one input and one output in Python.

Here's the code:

import numpy as np

class NeuralNetwork(object):
    def __init__(self, update_coefficient):
        self.weight = np.random.randn()
        self.predicted_output = 0
        self.update_coefficient = update_coefficient

    def sigmoid(self, val):
        res = 1.0 / (1.0 + np.exp(-val))
        return res

    def sigmoid_derivative(self, val):
        res = val * (1 - val)
        return res

    def get_cost(self, input, expected_output):
        self.predicted_output = self.sigmoid(input * self.weight)
        res = 0.5 * ((expected_output - self.predicted_output) ** 2)
        return res

    def get_cost_derivative_wrt_weight(self, input, expected_output):
        cost_deriv = ((expected_output - self.predicted_output) * 
            self.sigmoid_derivative(self.predicted_output) *
        return cost_deriv

    def update_weight(self, input, expected_output):
        self.weight += (self.update_coefficient * 
                        self.get_cost_derivative_wrt_weight(input, expected_output))

nn = NeuralNetwork(1.0)

input = 3; predicted = 2

for var in range(1000):
    nn.get_cost(input, predicted)
    nn.update_weight(input, predicted)

What is puzzling me is that the predicted value from the network is converging, but it is converging to 1, rather than 2.

What am I doing wrong here?

  • $\begingroup$ Hi Lukasz Grad, thanks so much - I found that when I removed the sigmoid parts of the code, my network worked like a dream. Can you post your comment as an answer so that I can mark it as accepted? Also, is there a point to sigmoid functions in neural networks if you don't need the network to classify a set of inputs as either 0 or 1? $\endgroup$ – Tola Odejayi Apr 16 '17 at 22:03

Simple, sigmoid is a function $ℜ→(0,1)$, so $1$ is as far as it can get.

Follow-up on comment question:

Sigmoid function is important as it introduces non-linearity to the network, it greatly increases the space of functions that can be approximated.

Typically, we use sigmoids (or other non-linear functions) in hidden layers of network. Acitvation function in output layer is dependent on the domain of values we want to predict. It can simply be identity if the output is in whole $\Re$ set or sigmoid if we need e.g. probability estimates in $[0,1]$ range. Hope it helps


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.