I am attempting to implement a back propagation algorithm that can efficevley read from a file of features and there targets and predict there outputs correctly, however for sake of testing I am hard coding the inputs (features and target). I am using this pseudocode for reference:
My code
import math
# Features
x = [[0,0,1], [0,1,1], [1,0,1]]
# Random small weights
w = [0.5, -0.1, 0.2]
# (weight change)
delta_w = [0,0,0]
# Target and learning rate
learning_rate = 0.05
t = [0,0,1]
def get_output(x):
net = 0
# Calculate the net input
for i in range(0, len(x)):
net += x[i] * w[i]
# Activation function
output = 1 / (1 + math.exp(-net))
return output
def update_delta_w(t, output, x):
for i in range(0, len(delta_w)):
delta_w[i] += learning_rate * (t - output) * x[i]
def update_weights():
for i in range(0, len(w)):
w[i] = delta_w[i] + w[i]
if __name__ == "__main__":
iteration_amount = 100
for episode in range(0, iteration_amount):
for example in x:
output = get_output(example)
print(f"Error: {str(t[x.index(example)] - output)}")
update_delta_w(t[x.index(example)], output, example)
update_weights()
print("New weights:")
print(', '.join(str(round(x, 3)) for x in w))
However, I am noticing my error rate isn't decreasing and my final weights increase with more iterations. I have attempted to play around with the learning rate, this doesn't have much effectiveness so I stuck with 0.05, and also double-checked my formulas and they seem to be done correctly from what I can see (I could be wrong).
I am quite new to neutral networks, so, unfortunately, struggling to correctly debug what I have done wrong. If anyone knows what I am doing wrong / has any tips would be great. Thank you.