I'm building some simple Perceptron networks to gain insight into how they operate. Most of the results are compelling, but there is one that I cannot figure out.
Here is my simple Perceptron class that I'll use to instantiate the nodes. Given the number of inputs, it will uniformly randomly select weights between -1 and +1. The default activation is sigmoid. The forward pass is easy, but the backward pass takes a little explanation: the error that is passed in is used to adjust the weights, but then a list
back_error is returned. Each element in the list corresponds to the amount of error to be propagated back along that input path. A multi-layer network would use
back_error to propagate the error backwards to previous layers.
class Perceptron: def __init__(self, num_inputs, act='sigmoid'): self.weights =  self.num_inputs = num_inputs self.act = act # define activation function with sigmoid being the default for x in range(0, num_inputs): self.weights.append(random.random() * 2 - 1) print(self.weights) def get_weights(self): return self.weights def feed_forward(self, inputs): self.inputs = inputs sum = 0 # multiply inputs by weights and sum them for i in range(0, self.num_inputs): sum += self.weights[i] * inputs[i] # 'activate' the sum and get the derivative self.output, self.output_prime = self.activate(sum) return self.output def activate(self, x): if (self.act == 'sigmoid'): activation = self.sigmoid(x) activation_prime = activation * (1 - activation) else: activation = self.step(x) activation_prime = 1 # use 1 since step activation is not differentiable return activation, activation_prime def sigmoid(self, x): return 1/(1 + np.exp(-x)) def step(self, x): if x > 0: return 1 return 0 def backward_pass(self, error): learning_rate = 0.01 # hyperparameter back_error =  # each element in list represent amount of error to send backward along that connection for i in range(0, self.num_inputs): back_error.append(error * self.output_prime * self.weights[i]) self.weights[i] -= error * self.output_prime * self.inputs[i] * learning_rate return back_error
I built a number of networks with this class (e.g. single-node classier for point above a line, single-node AND and OR, two-node XOR, three-node XOR, etc.) and they all work great. To get a sense of how it works in a more non-linear situation, I decided to build a 3-node network (2 hidden and 1 output) to attempt to determine if a given point is above a parabola.
def parabola(x): return 0.005 * pow(x - 500, 2) + 250 a = Perceptron(3, act='sigmoid') b = Perceptron(3, act='sigmoid') c = Perceptron(3, act='sigmoid') def network(first, second): a_out = a.feed_forward([first, second, 1]) b_out = b.feed_forward([first, second, 1]) c_out = c.feed_forward([a_out, b_out, 1]) return c_out
Initially I had ~30% accuracy (prior to training), but after training with a couple million synthesized data points I got over 98%, so it's working.
for _ in range(0,1000000): x_coord = random.random() * 1000 y_coord = random.random() * 1000 curve_y = parabola(x_coord) x_norm = x_coord / 1000 y_norm = y_coord / 1000 c_out = network(x_norm, y_norm) if curve_y > y_coord: answer = 1 else: answer = 0 back_error = c.backward_pass(c_out - answer) a.backward_pass(back_error) b.backward_pass(back_error) nodea = a.get_weights() print(nodea) print(-nodea/nodea, -nodea/nodea) nodeb = b.get_weights() print(nodeb) print(-nodeb/nodeb, -nodeb/nodeb) print(c.get_weights()) # determine the accuracy correct = 0 for _ in range(0,1000): x_coord = random.random() * 1000 y_coord = random.random() * 1000 curve_y = parabola(x_coord) x_norm = x_coord / 1000 y_norm = y_coord / 1000 is_above = curve_y > y_coord guess_above = network(x_norm, y_norm) if (is_above == True and guess_above >= 0.5): correct += 1 if (is_above == False and guess_above < 0.5): correct += 1 print(correct)
I printed out the weights in the middle because I want to investigate what the different nodes 'learned'. Here is the output:
[-10.50207080737064, -4.0929731838230365, 4.994772993198401] -2.565878234648289 1.2203287851822766 [9.98993436672944, -4.141613082123893, -4.9468089789837775] 2.412087794933858 -1.1944160115620859 [11.02140782709833, 10.58641375149512, -5.464087751867745]
The inputs to the hidden nodes are the x-coord of the point to test, the y-coord of said point and the bias of 1. Looking at the weights of node
a, the hypothesis is
-10.50x -4.09y + 4.99 = 0. Solving for
y (I had Python help - line #2), and then de-normalizing, we get
y = -2.57x + 1220. Doing the same for node
b we get
y = 2.41x - 1194.
This is awesome, because when I graph those two lines next to the original parabola, I get this:
Automagically, one hidden node 'learns' a line to the left of the parabola and the other hidden node 'learns' another line on the opposite side of the parabola. This is perfect, since the output of node
a should be >> 0.5 if the point is above its line while the output of node
b should be >> 0.5 if the point is above its line.
One would then think that the output node (node
c), which has for inputs the outputs of nodes
b along with a bias of 1, would be the logical AND of the outputs of those hidden nodes. In other words, if the given point is above node
a's line AND above node
b's line, then it's above the parabola. This, however, is not the case.
print('%f' % c.feed_forward([1, 1, 1])) print('%f' % c.feed_forward([1, 0, 1])) print('%f' % c.feed_forward([0, 1, 1])) print('%f' % c.feed_forward([0, 0, 1])) 1.000000 0.996156 0.994073 0.004218
The output node (node
c) 'learned' the logical OR from the outputs of the hidden nodes. This makes no sense whatsoever. From the graph one would clearly expect the output node to take the logical AND of the hidden nodes.
Could anyone please explain what is happening here? Thank you.