Unexpected weights in Gradient descent algorithm (linear classification) in python

Question

I am attempting to implement a back propagation algorithm that can efficevley read from a file of features and there targets and predict there outputs correctly, however for sake of testing I am hard coding the inputs (features and target). I am using this pseudocode for reference:

My code

import math

# Features
x = [[0,0,1], [0,1,1], [1,0,1]]

# Random small weights
w = [0.5, -0.1, 0.2]

# (weight change)
delta_w = [0,0,0]

# Target and learning rate
learning_rate = 0.05
t = [0,0,1]


def get_output(x):
    net = 0
    
    # Calculate the net input
    for i in range(0, len(x)):
        net += x[i] * w[i]
    
    # Activation function
    output =  1 / (1 + math.exp(-net))
    
    return output

def update_delta_w(t, output, x):
    for i in range(0, len(delta_w)):
        delta_w[i] += learning_rate * (t - output) * x[i]

def update_weights():
    for i in range(0, len(w)):
        w[i] = delta_w[i] + w[i]
        
        
if __name__ == "__main__":
    
    iteration_amount = 100
    
    for episode in range(0, iteration_amount):
        for example in x:
            output = get_output(example)
            
            print(f"Error: {str(t[x.index(example)] - output)}")
            
            update_delta_w(t[x.index(example)], output, example)

        update_weights()
    
    print("New weights:")
    print(', '.join(str(round(x, 3)) for x in w))

However, I am noticing my error rate isn't decreasing and my final weights increase with more iterations. I have attempted to play around with the learning rate, this doesn't have much effectiveness so I stuck with 0.05, and also double-checked my formulas and they seem to be done correctly from what I can see (I could be wrong).

I am quite new to neutral networks, so, unfortunately, struggling to correctly debug what I have done wrong. If anyone knows what I am doing wrong / has any tips would be great. Thank you.

Answer 1

Your gradients and updates are correct, and the error is decreasing in magnitude. It's in the order of $10^{-21}$ sometimes, e.g. 1.31231234e-21. It's a bit technicality but you should print the magnitude (or the square) of the error though.

The dataset is linearly separable, and there is no unique solution (I mean after thresholding). For example, if $[a,b,c]$ is a solution, then $[2a,2b,2c]$ is also a solution. That is why your weights do not stop increasing as you run more iterations. As the numbers increase, output values get close to sigmoid boundaries (i.e. 0 or 1).

You can add a regularization term to prevent this or stop somewhere after the samples are all classified correctly. In the end, you did not set a "termination condition" as described in the algorithm pseudocode. This can be for example "Error < 1e-5".