2
$\begingroup$

I am writing a program to perform a linear regression. The actual function I am trying to estimate (for testing purposes) is as follows:

f(x) = 3x + 100

The function I am using to estimate is as follows:

r(x) = wx + b

Where w and b are the parameters being adjusted using stochastic gradient descent. I use the mean squared error function as my loss function:

l = (f(x) - r(x))^2

So, the derivative of l w.r.t. w:

-2x(f(x)-xw-b)

and the derivative of l w.r.t. b:

-2(f(x)-xw-b)

So, to modify the parameters w and b I apply the following operations for every prediction:

w -= -2x(f(x)-xw-b) * 0.000001
b -= -2(f(x)-xw-b) * 0.000001

The value of w eventually converges on 3 (the value found in f(x)) but b seems to just hang around the number it starts on. Am I doing the math wrong?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ Math looks ok to me. You should probably say 'stochastic gradient descent', since you're operating on a single point at a time. SGD is sensitive to the learning rate. Yours is small, so maybe it's just progressing very slowly. Have you tried changing it to see what happens? How are you initializing the parameters, and generating x/y? What does your error look like? Since you don't have any noise, it should converge to 0. $\endgroup$
    – user20160
    Commented May 27, 2016 at 7:01
  • $\begingroup$ @user20160 Great call! The learning rate was too low for the bias. I was keeping the learning rate the same between both w and b. So when I increased the learning rate the whole thing diverged. But if I just increase the learning rate of b and keep w the same I now get a really close approximation. One quick question: Do learning rates always have to be tinkered with or is there some sort of intuition for determining useful learning rates? $\endgroup$
    – dobafresh
    Commented May 27, 2016 at 7:24
  • 1
    $\begingroup$ Another point to consider is the way the updates are written. You're updating w first, which changes the function. You then update b, but this update depends on the value of w, which has changed. So, this is more like 'coordinate descent' than gradient descent. I think it should still converge in this case, but possibly less efficient (?) $\endgroup$
    – user20160
    Commented May 27, 2016 at 7:30
  • 1
    $\begingroup$ @dobrafresh Glad it worked. In general, SGD learning rates are a hyperparameter and have to be tinkered with. There are various schemes for adjusting them, which you can find in the neural networks literature. An interesting approach is to use separate learning rates for each parameter (as you've discovered), and to adjust them adaptively. There are some very successful methods that do this. Check out RMSProp, Adadelta, Adam. They have hyperparameters too, but they're less sensitive than SGD learning rates. $\endgroup$
    – user20160
    Commented May 27, 2016 at 7:33

1 Answer 1

Reset to default
2
$\begingroup$

I came across the same problem recently. The linear regression result diverges with learning rate = 0.1. I keep decreasing the learning rate until it reached 1e-5 until the linear regression converges. Learning rate is critical for linear regression. If it is too small, it will slow the calculation. If it is too big, it will overshoot the minimum of the cost function or even diverge. The other way to solve this problem is to start with better w and b values. Take a look at the data, and start with decent w and b.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.