# Large Feature Values for Gradient Descent

Recently, I work on a linear regression model of my project. I have 200 samples, each of which has only one feature, to train my model. When I try to apply gradient descent algorithm I cannot reach optimum parameter value even if I choose small step rate, which is equal to 0.001.

When I choose this step value, computed parameter values always cross the other side. I tried to plot how gradient descent algorithm behaves like.

    \        /
w1<- \*<---*/ -> w0
\    /
\__/


I thought the reason a little bit, and I concluded that main reason is the fact that feature values are very big. Feature values and ground truth labels are around 7000-8000.

Is there a solution to such a problem ? The feature values are from cm^3 unit and ground truth labels are from gram unit. I think that I can convert them into dm^3 and kg, but if I do that, I think that I may cause domain shift problem.

• Is this the smallest learning rate you tried? Maybe try things like 1e-6 etc and see what happens..? – Tim Mar 22 at 16:41
• If it's really just a linear regression with 200 samples, there is a method for computing exact solutions by solving a system of linear equations. There's rarely a need to use gradient descent in a situation like this. – Matthew Drury Mar 22 at 16:59
• I know using a basic linear system is a solution, but this is a general problem about gradient descent. Maybe using such a system can solve my problem, but the people who need to use gradient descent for large dataset and have big feature values need a solution to this problem. @MatthewDrury – Goktug Mar 22 at 17:30
• @Goktug Fair enough! – Matthew Drury Mar 22 at 17:36
• Maybe the learning rate ist too small? Maybe you could use a different algorithm which modifiest the learning rate adaptively like ADAM? – Karsten W. Mar 22 at 18:40