2
$\begingroup$

I'm building some simple Perceptron networks to gain insight into how they operate. Most of the results are compelling, but there is one that I cannot figure out.

Here is my simple Perceptron class that I'll use to instantiate the nodes. Given the number of inputs, it will uniformly randomly select weights between -1 and +1. The default activation is sigmoid. The forward pass is easy, but the backward pass takes a little explanation: the error that is passed in is used to adjust the weights, but then a list back_error[] is returned. Each element in the list corresponds to the amount of error to be propagated back along that input path. A multi-layer network would use back_error[] to propagate the error backwards to previous layers.

class Perceptron:

    def __init__(self, num_inputs, act='sigmoid'):
        self.weights = []
        self.num_inputs = num_inputs
        self.act = act # define activation function with sigmoid being the default
        for x in range(0, num_inputs):
            self.weights.append(random.random() * 2 - 1)
        print(self.weights)

    def get_weights(self):
        return self.weights

    def feed_forward(self, inputs):
        self.inputs = inputs
        sum = 0

        # multiply inputs by weights and sum them
        for i in range(0, self.num_inputs):
            sum += self.weights[i] * inputs[i]

        # 'activate' the sum and get the derivative
        self.output, self.output_prime = self.activate(sum)
        return self.output

    def activate(self, x):
        if (self.act == 'sigmoid'):
            activation = self.sigmoid(x)
            activation_prime = activation * (1 - activation)
        else:
            activation = self.step(x)
            activation_prime = 1 # use 1 since step activation is not differentiable
        return activation, activation_prime

    def sigmoid(self, x):
        return 1/(1 + np.exp(-x))

    def step(self, x):
        if x > 0:
            return 1
        return 0

    def backward_pass(self, error):
        learning_rate = 0.01 # hyperparameter
        back_error = [] # each element in list represent amount of error to send backward along that connection
        for i in range(0, self.num_inputs):
            back_error.append(error * self.output_prime * self.weights[i])
            self.weights[i] -= error * self.output_prime * self.inputs[i] * learning_rate
        return back_error

I built a number of networks with this class (e.g. single-node classier for point above a line, single-node AND and OR, two-node XOR, three-node XOR, etc.) and they all work great. To get a sense of how it works in a more non-linear situation, I decided to build a 3-node network (2 hidden and 1 output) to attempt to determine if a given point is above a parabola.

def parabola(x):
    return 0.005 * pow(x - 500, 2) + 250

a = Perceptron(3, act='sigmoid')
b = Perceptron(3, act='sigmoid')
c = Perceptron(3, act='sigmoid')

def network(first, second):
    a_out = a.feed_forward([first, second, 1])
    b_out = b.feed_forward([first, second, 1])
    c_out = c.feed_forward([a_out, b_out, 1])
    return c_out

Initially I had ~30% accuracy (prior to training), but after training with a couple million synthesized data points I got over 98%, so it's working.

for _ in range(0,1000000):
    x_coord = random.random() * 1000
    y_coord = random.random() * 1000
    curve_y = parabola(x_coord)
    x_norm = x_coord / 1000
    y_norm = y_coord / 1000

    c_out = network(x_norm, y_norm)

    if curve_y > y_coord:
        answer = 1
    else:
        answer = 0

    back_error = c.backward_pass(c_out - answer)
    a.backward_pass(back_error[0])
    b.backward_pass(back_error[1])

nodea = a.get_weights()
print(nodea)
print(-nodea[0]/nodea[1], -nodea[2]/nodea[1])
nodeb = b.get_weights()
print(nodeb)
print(-nodeb[0]/nodeb[1], -nodeb[2]/nodeb[1])
print(c.get_weights())

# determine the accuracy
correct = 0

for _ in range(0,1000):
    x_coord = random.random() * 1000
    y_coord = random.random() * 1000
    curve_y = parabola(x_coord)
    x_norm = x_coord / 1000
    y_norm = y_coord / 1000

    is_above = curve_y > y_coord
    guess_above = network(x_norm, y_norm)

    if (is_above == True and guess_above >= 0.5):
        correct += 1
    if (is_above == False and guess_above < 0.5):
        correct += 1

print(correct)

I printed out the weights in the middle because I want to investigate what the different nodes 'learned'. Here is the output:

[-10.50207080737064, -4.0929731838230365, 4.994772993198401]
-2.565878234648289 1.2203287851822766
[9.98993436672944, -4.141613082123893, -4.9468089789837775]
2.412087794933858 -1.1944160115620859
[11.02140782709833, 10.58641375149512, -5.464087751867745]

The inputs to the hidden nodes are the x-coord of the point to test, the y-coord of said point and the bias of 1. Looking at the weights of node a, the hypothesis is -10.50x -4.09y + 4.99 = 0. Solving for y (I had Python help - line #2), and then de-normalizing, we get y = -2.57x + 1220. Doing the same for node b we get y = 2.41x - 1194.

This is awesome, because when I graph those two lines next to the original parabola, I get this:

Parabola with two hidden nodes

Automagically, one hidden node 'learns' a line to the left of the parabola and the other hidden node 'learns' another line on the opposite side of the parabola. This is perfect, since the output of node a should be >> 0.5 if the point is above its line while the output of node b should be >> 0.5 if the point is above its line.

One would then think that the output node (node c), which has for inputs the outputs of nodes a and b along with a bias of 1, would be the logical AND of the outputs of those hidden nodes. In other words, if the given point is above node a's line AND above node b's line, then it's above the parabola. This, however, is not the case.

print('%f' % c.feed_forward([1, 1, 1]))
print('%f' % c.feed_forward([1, 0, 1]))
print('%f' % c.feed_forward([0, 1, 1]))
print('%f' % c.feed_forward([0, 0, 1]))

1.000000
0.996156
0.994073
0.004218

The output node (node c) 'learned' the logical OR from the outputs of the hidden nodes. This makes no sense whatsoever. From the graph one would clearly expect the output node to take the logical AND of the hidden nodes.

Could anyone please explain what is happening here? Thank you.

$\endgroup$
  • $\begingroup$ This does seem very strange and it seems to contradict the 98% accuracy you achieve as any points in the bottom left corner should be categorized badly. Here are my two best guesses: 1. maybe your network learned that it was able to correctly categorize most of the input with the OR-rule and failed to improve the score due to the limitation of the sigmoid function to properly model the non-linearity. $\endgroup$ – GR4 Apr 16 '18 at 12:29
  • 1
    $\begingroup$ 2. Perhaps your c node learned something more complex than a simple OR. What happens when you feed intermediate input eg [0.6, 0.4, 1] etc. If guess 2 is correct, then the 2% wrong categorizations might be occurring near the outer limits of your data. $\endgroup$ – GR4 Apr 16 '18 at 12:55
  • $\begingroup$ I'm getting closer. For a reason I haven't figured out yet, a and b are in fact learning if the point is under the line (I wrote some code and took a peek), which is surprising. Regardless, if c is then taking the OR of those outputs, it's learning the entire space below the parabola. That's great, but I'm trying to teach it the opposite of that. $\endgroup$ – Mark Cramer Apr 17 '18 at 0:44
  • $\begingroup$ I have figured it out. The comment above indicates a problem of polarity, so during training and testing for accuracy, I need to switch to y_coord > curve_y. It now works perfectly. That's a lot of question for a silly error, so I'm not sure if I should leave this or not. Would be happy to give points to @GR4 if you'd like to post the answer. $\endgroup$ – Mark Cramer Apr 17 '18 at 1:41
  • $\begingroup$ Ow ok, don't worry about the points, it was a pleasure reading and thinking about your question. Not sure what the stackexchange policies are about keeping the question or not, but if i were you i would consider writing a small blogpost or so about your experiments - they provide some nice insights about the functionality of simple networks! $\endgroup$ – GR4 Apr 17 '18 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.