Recently, I work on a linear regression model of my project. I have 200 samples, each of which has only one feature, to train my model. When I try to apply gradient descent algorithm I cannot reach optimum parameter value even if I choose small step rate, which is equal to 0.001.

When I choose this step value, computed parameter values always cross the other side. I tried to plot how gradient descent algorithm behaves like.

    \        /
w1<- \*<---*/ -> w0
      \    /

I thought the reason a little bit, and I concluded that main reason is the fact that feature values are very big. Feature values and ground truth labels are around 7000-8000.

Is there a solution to such a problem ? The feature values are from cm^3 unit and ground truth labels are from gram unit. I think that I can convert them into dm^3 and kg, but if I do that, I think that I may cause domain shift problem.

  • $\begingroup$ Is this the smallest learning rate you tried? Maybe try things like 1e-6 etc and see what happens..? $\endgroup$ – Tim Mar 22 at 16:41
  • $\begingroup$ If it's really just a linear regression with 200 samples, there is a method for computing exact solutions by solving a system of linear equations. There's rarely a need to use gradient descent in a situation like this. $\endgroup$ – Matthew Drury Mar 22 at 16:59
  • 1
    $\begingroup$ I know using a basic linear system is a solution, but this is a general problem about gradient descent. Maybe using such a system can solve my problem, but the people who need to use gradient descent for large dataset and have big feature values need a solution to this problem. @MatthewDrury $\endgroup$ – Goktug Mar 22 at 17:30
  • $\begingroup$ @Goktug Fair enough! $\endgroup$ – Matthew Drury Mar 22 at 17:36
  • $\begingroup$ Maybe the learning rate ist too small? Maybe you could use a different algorithm which modifiest the learning rate adaptively like ADAM? $\endgroup$ – Karsten W. Mar 22 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.