import matplotlib.pyplot as plt inputs = [(0.0000, 0.0000), (0.1600, 0.1556), (0.2400, 0.3543), (0.2800, 0.3709)] targets = [230, 555, 815, 860] weights = [0.1, 0.2] b = 0.3 learning_rate = 0.1 epochs = 4 # prediction def predict(inputs): return sum([(w * i) for w, i in zip(weights, inputs)]) + b # train the network for epoch in range(epochs): # Feed forward--------- pred = [predict(inp) for inp in inputs] print("Pred:", pred) # Back propagation------ # error derivative errors_d = [(p - t) for p, t in zip(pred, targets)] # error partial derivative weight_d = [[(e * i) for i in (inp)] for e, inp in zip(errors_d, inputs)] bias_d = [(e * 1) for e in errors_d] weight_d_T = list(zip(*weight_d)) # Update weights and bias for j in range(len(weights)): weights[j] -= learning_rate * (sum(weight_d_T[j]) / len(weight_d)) b = b - (learning_rate * (sum(bias_d) / len(bias_d)))
From theory, In order to minimize error, we need to take the derivative with respect to the weights and bias. In the above code, I did partial derivation of the error function. And use it with learning rate and update weights and bias.
After doing some tests I figured out that if I write the weight updating equation as
weights[j] -= learning_rate * (sum(weight_d_T[j]) / len(weight_d)) it will move towards down of the slope. If I write the weight updating equation as
weights[j] += learning_rate * (sum(weight_d_T[j]) / len(weight_d)) I mean add partial derivative of error with previous weight value, then it will move upward of the slope. What I mean by moving upward or downward of the slope could be understood from the image below.
My question Q1: In my code, I explicitly didn't write any code on taking derivative of error equals to zero. How does the derivative reach the point where the error is zero?
Q2: After reaching the point where the error is zero or nearly zero how does the algorithm know that, it reached the point where the error becomes zero? I didn't write any if condition like if partial derivative of error == 0 then that's our target point. Without any, if statement it works. How?
I tried my best to make understand my question. But, if there is confusion please ask in the comments. Thank you.