# Unexpected weights in Gradient descent algorithm (linear classification) in python

I am attempting to implement a back propagation algorithm that can efficevley read from a file of features and there targets and predict there outputs correctly, however for sake of testing I am hard coding the inputs (features and target). I am using this pseudocode for reference:

My code

import math

# Features
x = [[0,0,1], [0,1,1], [1,0,1]]

# Random small weights
w = [0.5, -0.1, 0.2]

# (weight change)
delta_w = [0,0,0]

# Target and learning rate
learning_rate = 0.05
t = [0,0,1]

def get_output(x):
net = 0

# Calculate the net input
for i in range(0, len(x)):
net += x[i] * w[i]

# Activation function
output =  1 / (1 + math.exp(-net))

return output

def update_delta_w(t, output, x):
for i in range(0, len(delta_w)):
delta_w[i] += learning_rate * (t - output) * x[i]

def update_weights():
for i in range(0, len(w)):
w[i] = delta_w[i] + w[i]

if __name__ == "__main__":

iteration_amount = 100

for episode in range(0, iteration_amount):
for example in x:
output = get_output(example)

print(f"Error: {str(t[x.index(example)] - output)}")

update_delta_w(t[x.index(example)], output, example)

update_weights()

print("New weights:")
print(', '.join(str(round(x, 3)) for x in w))


However, I am noticing my error rate isn't decreasing and my final weights increase with more iterations. I have attempted to play around with the learning rate, this doesn't have much effectiveness so I stuck with 0.05, and also double-checked my formulas and they seem to be done correctly from what I can see (I could be wrong).

I am quite new to neutral networks, so, unfortunately, struggling to correctly debug what I have done wrong. If anyone knows what I am doing wrong / has any tips would be great. Thank you.

• – Sycorax
Commented Mar 6, 2022 at 18:14

Your gradients and updates are correct, and the error is decreasing in magnitude. It's in the order of $$10^{-21}$$ sometimes, e.g. 1.31231234e-21. It's a bit technicality but you should print the magnitude (or the square) of the error though.
The dataset is linearly separable, and there is no unique solution (I mean after thresholding). For example, if $$[a,b,c]$$ is a solution, then $$[2a,2b,2c]$$ is also a solution. That is why your weights do not stop increasing as you run more iterations. As the numbers increase, output values get close to sigmoid boundaries (i.e. 0 or 1).