Output node of Perceptron neural network 'learning' unexpected function

I'm building some simple Perceptron networks to gain insight into how they operate. Most of the results are compelling, but there is one that I cannot figure out.

Here is my simple Perceptron class that I'll use to instantiate the nodes. Given the number of inputs, it will uniformly randomly select weights between -1 and +1. The default activation is sigmoid. The forward pass is easy, but the backward pass takes a little explanation: the error that is passed in is used to adjust the weights, but then a list back_error[] is returned. Each element in the list corresponds to the amount of error to be propagated back along that input path. A multi-layer network would use back_error[] to propagate the error backwards to previous layers.

class Perceptron:

def __init__(self, num_inputs, act='sigmoid'):
self.weights = []
self.num_inputs = num_inputs
self.act = act # define activation function with sigmoid being the default
for x in range(0, num_inputs):
self.weights.append(random.random() * 2 - 1)
print(self.weights)

def get_weights(self):
return self.weights

def feed_forward(self, inputs):
self.inputs = inputs
sum = 0

# multiply inputs by weights and sum them
for i in range(0, self.num_inputs):
sum += self.weights[i] * inputs[i]

# 'activate' the sum and get the derivative
self.output, self.output_prime = self.activate(sum)
return self.output

def activate(self, x):
if (self.act == 'sigmoid'):
activation = self.sigmoid(x)
activation_prime = activation * (1 - activation)
else:
activation = self.step(x)
activation_prime = 1 # use 1 since step activation is not differentiable
return activation, activation_prime

def sigmoid(self, x):
return 1/(1 + np.exp(-x))

def step(self, x):
if x > 0:
return 1
return 0

def backward_pass(self, error):
learning_rate = 0.01 # hyperparameter
back_error = [] # each element in list represent amount of error to send backward along that connection
for i in range(0, self.num_inputs):
back_error.append(error * self.output_prime * self.weights[i])
self.weights[i] -= error * self.output_prime * self.inputs[i] * learning_rate
return back_error

I built a number of networks with this class (e.g. single-node classier for point above a line, single-node AND and OR, two-node XOR, three-node XOR, etc.) and they all work great. To get a sense of how it works in a more non-linear situation, I decided to build a 3-node network (2 hidden and 1 output) to attempt to determine if a given point is above a parabola.

def parabola(x):
return 0.005 * pow(x - 500, 2) + 250

a = Perceptron(3, act='sigmoid')
b = Perceptron(3, act='sigmoid')
c = Perceptron(3, act='sigmoid')

def network(first, second):
a_out = a.feed_forward([first, second, 1])
b_out = b.feed_forward([first, second, 1])
c_out = c.feed_forward([a_out, b_out, 1])
return c_out

Initially I had ~30% accuracy (prior to training), but after training with a couple million synthesized data points I got over 98%, so it's working.

for _ in range(0,1000000):
x_coord = random.random() * 1000
y_coord = random.random() * 1000
curve_y = parabola(x_coord)
x_norm = x_coord / 1000
y_norm = y_coord / 1000

c_out = network(x_norm, y_norm)

if curve_y > y_coord:
else:

a.backward_pass(back_error[0])
b.backward_pass(back_error[1])

nodea = a.get_weights()
print(nodea)
print(-nodea[0]/nodea[1], -nodea[2]/nodea[1])
nodeb = b.get_weights()
print(nodeb)
print(-nodeb[0]/nodeb[1], -nodeb[2]/nodeb[1])
print(c.get_weights())

# determine the accuracy
correct = 0

for _ in range(0,1000):
x_coord = random.random() * 1000
y_coord = random.random() * 1000
curve_y = parabola(x_coord)
x_norm = x_coord / 1000
y_norm = y_coord / 1000

is_above = curve_y > y_coord
guess_above = network(x_norm, y_norm)

if (is_above == True and guess_above >= 0.5):
correct += 1
if (is_above == False and guess_above < 0.5):
correct += 1

print(correct)

I printed out the weights in the middle because I want to investigate what the different nodes 'learned'. Here is the output:

[-10.50207080737064, -4.0929731838230365, 4.994772993198401]
-2.565878234648289 1.2203287851822766
[9.98993436672944, -4.141613082123893, -4.9468089789837775]
2.412087794933858 -1.1944160115620859
[11.02140782709833, 10.58641375149512, -5.464087751867745]

The inputs to the hidden nodes are the x-coord of the point to test, the y-coord of said point and the bias of 1. Looking at the weights of node a, the hypothesis is -10.50x -4.09y + 4.99 = 0. Solving for y (I had Python help - line #2), and then de-normalizing, we get y = -2.57x + 1220. Doing the same for node b we get y = 2.41x - 1194.

This is awesome, because when I graph those two lines next to the original parabola, I get this:

Automagically, one hidden node 'learns' a line to the left of the parabola and the other hidden node 'learns' another line on the opposite side of the parabola. This is perfect, since the output of node a should be >> 0.5 if the point is above its line while the output of node b should be >> 0.5 if the point is above its line.

One would then think that the output node (node c), which has for inputs the outputs of nodes a and b along with a bias of 1, would be the logical AND of the outputs of those hidden nodes. In other words, if the given point is above node a's line AND above node b's line, then it's above the parabola. This, however, is not the case.

print('%f' % c.feed_forward([1, 1, 1]))
print('%f' % c.feed_forward([1, 0, 1]))
print('%f' % c.feed_forward([0, 1, 1]))
print('%f' % c.feed_forward([0, 0, 1]))

1.000000
0.996156
0.994073
0.004218

The output node (node c) 'learned' the logical OR from the outputs of the hidden nodes. This makes no sense whatsoever. From the graph one would clearly expect the output node to take the logical AND of the hidden nodes.

Could anyone please explain what is happening here? Thank you.

• This does seem very strange and it seems to contradict the 98% accuracy you achieve as any points in the bottom left corner should be categorized badly. Here are my two best guesses: 1. maybe your network learned that it was able to correctly categorize most of the input with the OR-rule and failed to improve the score due to the limitation of the sigmoid function to properly model the non-linearity. – GR4 Apr 16 '18 at 12:29
• 2. Perhaps your c node learned something more complex than a simple OR. What happens when you feed intermediate input eg [0.6, 0.4, 1] etc. If guess 2 is correct, then the 2% wrong categorizations might be occurring near the outer limits of your data. – GR4 Apr 16 '18 at 12:55
• I'm getting closer. For a reason I haven't figured out yet, a and b are in fact learning if the point is under the line (I wrote some code and took a peek), which is surprising. Regardless, if c is then taking the OR of those outputs, it's learning the entire space below the parabola. That's great, but I'm trying to teach it the opposite of that. – Mark Cramer Apr 17 '18 at 0:44
• I have figured it out. The comment above indicates a problem of polarity, so during training and testing for accuracy, I need to switch to y_coord > curve_y. It now works perfectly. That's a lot of question for a silly error, so I'm not sure if I should leave this or not. Would be happy to give points to @GR4 if you'd like to post the answer. – Mark Cramer Apr 17 '18 at 1:41
• Ow ok, don't worry about the points, it was a pleasure reading and thinking about your question. Not sure what the stackexchange policies are about keeping the question or not, but if i were you i would consider writing a small blogpost or so about your experiments - they provide some nice insights about the functionality of simple networks! – GR4 Apr 17 '18 at 7:44