Previous questions this & this does not answer my question


import matplotlib.pyplot as plt

inputs = [(0.0000, 0.0000), (0.1600, 0.1556), (0.2400, 0.3543), (0.2800, 0.3709)]
targets = [230, 555, 815, 860]

weights = [0.1, 0.2]
b = 0.3
learning_rate = 0.1

epochs = 4

# prediction
def predict(inputs):
    return sum([(w * i) for w, i in zip(weights, inputs)]) + b

# train the network
for epoch in range(epochs):
#   Feed forward--------- 
    pred = [predict(inp) for inp in inputs]
    print("Pred:", pred)
#   Back propagation------

#   error derivative
    errors_d = [(p - t) for p, t in zip(pred, targets)]
#   error partial derivative
    weight_d = [[(e * i) for i in (inp)] for e, inp in zip(errors_d, inputs)]
    bias_d = [(e * 1) for e in errors_d]

    weight_d_T = list(zip(*weight_d))
#   Update weights and bias
    for j in range(len(weights)):
        weights[j] -= learning_rate * (sum(weight_d_T[j]) / len(weight_d))
    b = b - (learning_rate * (sum(bias_d) / len(bias_d)))

From theory, In order to minimize error, we need to take the derivative with respect to the weights and bias. In the above code, I did partial derivation of the error function. And use it with learning rate and update weights and bias.

After doing some tests I figured out that if I write the weight updating equation as weights[j] -= learning_rate * (sum(weight_d_T[j]) / len(weight_d)) it will move towards down of the slope. If I write the weight updating equation as weights[j] += learning_rate * (sum(weight_d_T[j]) / len(weight_d)) I mean add partial derivative of error with previous weight value, then it will move upward of the slope. What I mean by moving upward or downward of the slope could be understood from the image below.

error function

My question Q1: In my code, I explicitly didn't write any code on taking derivative of error equals to zero. How does the derivative reach the point where the error is zero?

Q2: After reaching the point where the error is zero or nearly zero how does the algorithm know that, it reached the point where the error becomes zero? I didn't write any if condition like if partial derivative of error == 0 then that's our target point. Without any, if statement it works. How?

I tried my best to make understand my question. But, if there is confusion please ask in the comments. Thank you.


1 Answer 1


Q1: weight_d is the derivative that you are calculating... the point is that is na iterative minimization procedure, thus you never set your equation equal to 0, but you slowing go down the slope, iteration per iteration... imagine you are on top a mountain, you don't jump to the lowest point, you do a step at a time where the gravity brings you

You actually can't know, because it's an optimization algorithm... GD guarantees to converge with infinite steps... however if you have some conditions, for example strong convexity, you have an upper bound on the distance from the global minimum (for convex optimization problems)

  • $\begingroup$ From your answer what I understood is calculating weight_d in several epochs gradually moves towards zero. But, I can't see how it's moving towards it. It just moves in that way. I didn't understand the answer to the 2nd question though. $\endgroup$
    – F.C. Akhi
    Sep 7, 2022 at 16:04
  • $\begingroup$ @F.C.Akhi at each step you find a direction which decreases the function (negative derivative), and you do a step in that direction... to this enough time, and you get very close to the minimum $\endgroup$
    – Alberto
    Sep 8, 2022 at 12:52
  • $\begingroup$ @F.C.Akhi with respect to the second question... you can't generally know how close you are to the actual minimum... however, sometimes you can get an upper bound on that distance (current calculated solution - actual minimum) $\endgroup$
    – Alberto
    Sep 8, 2022 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.