I'm doing some machine learning stuff. And stumbled upon the machine learning regression algorithm. Here the derivatives are the ones from MSE:

$$f(m,b) = \frac{1}{N} \sum_{i=1}^{n} (y_i - (mx_i + b))^2$$

If you know the algorithm, jump off the code to the question. But this is an example:

def update_weights(radio, sales, weight, bias, learning_rate):
    weight_deriv = 0
    bias_deriv = 0
    companies = len(radio)

    for i in range(companies):
        # Calculate partial derivatives
        # -2x(y - (mx + b))
        weight_deriv += -2*radio[i] * (sales[i] - (weight*radio[i] + bias))

        # -2(y - (mx + b))
        bias_deriv += -2*(sales[i] - (weight*radio[i] + bias))

    # We subtract because the derivatives point in direction of steepest ascent
    weight -= (weight_deriv / companies) * learning_rate
    bias -= (bias_deriv / companies) * learning_rate
    return weight, bias

But my trouble is with this 2 lines:

weight -= (weight_deriv / companies) * learning_rate
bias -= (bias_deriv / companies) * learning_rate

I get they are slopes in Error(weight, bias)

Why do we update the parameters like that?

Once we have the direction we need to move, why not to use

weight -= learning_step

This looked a bit complicated to me I tried $E = X^2$, and $\large\frac{dE}{dX}=2X$

So to move from a particular X to one where E is smaller I could just use X-dX so I go anywhere.

Please just help don't answer!!

  • 1
    $\begingroup$ Do you know how gradient descent works? We're just minimizing $f$ using gradient descent. At each iteration, you take a step in the direction of steepest descent, which is the negative gradient direction. You could read about the gradient, gradient descent, and how to compute the gradient of $f$. $\endgroup$
    – littleO
    Dec 20, 2020 at 16:54
  • $\begingroup$ i cant get conceptually why the new W value is calculated with the derivative @littleO $\endgroup$
    – Minsky
    Dec 20, 2020 at 16:56
  • $\begingroup$ $ w -= df/dw*learning $ is wrong to me. This gives me a new point in $F$ $\endgroup$
    – Minsky
    Dec 20, 2020 at 16:57
  • 1
    $\begingroup$ But have you studied gradient descent? You have to learn about gradient descent in order to understand this. $\endgroup$
    – littleO
    Dec 20, 2020 at 17:01
  • $\begingroup$ ml-cheatsheet.readthedocs.io/en/latest/linear_regression.html $\endgroup$
    – Minsky
    Dec 20, 2020 at 17:04


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.