3
$\begingroup$

I am attempting to implement a back propagation algorithm that can efficevley read from a file of features and there targets and predict there outputs correctly, however for sake of testing I am hard coding the inputs (features and target). I am using this pseudocode for reference:

enter image description here

My code

import math

# Features
x = [[0,0,1], [0,1,1], [1,0,1]]

# Random small weights
w = [0.5, -0.1, 0.2]

# (weight change)
delta_w = [0,0,0]

# Target and learning rate
learning_rate = 0.05
t = [0,0,1]


def get_output(x):
    net = 0
    
    # Calculate the net input
    for i in range(0, len(x)):
        net += x[i] * w[i]
    
    # Activation function
    output =  1 / (1 + math.exp(-net))
    
    return output

def update_delta_w(t, output, x):
    for i in range(0, len(delta_w)):
        delta_w[i] += learning_rate * (t - output) * x[i]

def update_weights():
    for i in range(0, len(w)):
        w[i] = delta_w[i] + w[i]
        
        
if __name__ == "__main__":
    
    iteration_amount = 100
    
    for episode in range(0, iteration_amount):
        for example in x:
            output = get_output(example)
            
            print(f"Error: {str(t[x.index(example)] - output)}")
            
            update_delta_w(t[x.index(example)], output, example)

        update_weights()
    
    print("New weights:")
    print(', '.join(str(round(x, 3)) for x in w))

However, I am noticing my error rate isn't decreasing and my final weights increase with more iterations. I have attempted to play around with the learning rate, this doesn't have much effectiveness so I stuck with 0.05, and also double-checked my formulas and they seem to be done correctly from what I can see (I could be wrong).

I am quite new to neutral networks, so, unfortunately, struggling to correctly debug what I have done wrong. If anyone knows what I am doing wrong / has any tips would be great. Thank you.

$\endgroup$
1

1 Answer 1

1
$\begingroup$

Your gradients and updates are correct, and the error is decreasing in magnitude. It's in the order of $10^{-21}$ sometimes, e.g. 1.31231234e-21. It's a bit technicality but you should print the magnitude (or the square) of the error though.

The dataset is linearly separable, and there is no unique solution (I mean after thresholding). For example, if $[a,b,c]$ is a solution, then $[2a,2b,2c]$ is also a solution. That is why your weights do not stop increasing as you run more iterations. As the numbers increase, output values get close to sigmoid boundaries (i.e. 0 or 1).

You can add a regularization term to prevent this or stop somewhere after the samples are all classified correctly. In the end, you did not set a "termination condition" as described in the algorithm pseudocode. This can be for example "Error < 1e-5".

$\endgroup$
3
  • 1
    $\begingroup$ +1 Since the data are entirely fabricated, one way to have an identified model is to use fabricated data that are not linear separable. $\endgroup$
    – Sycorax
    Commented Mar 6, 2022 at 18:12
  • $\begingroup$ Thanks for the answer @gunes , so you are suggesting I can keep the iteration amount but also have another condition e.g. while iteration_amount > 0 or error_rate < 0.00001 and I calculate error_rate by error_rate = t[x.index(example)] - output, is this correct? $\endgroup$ Commented Mar 6, 2022 at 22:18
  • $\begingroup$ Yes, but it should be the absolute value of the average (or sum) error. Not a single sample. $\endgroup$
    – gunes
    Commented Mar 7, 2022 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.