# How To Choose Step Size (Learning Rate) in Batch Gradient Descent?

I'm practicing machine learning in python and trying to implement batch gradient descent algorithm by myself. Mathematically algorith is defined as follows:

$\theta_j = \theta_j + \alpha \sum_{i=0}^{n}{(y^{(i)}-h_{\theta}^{(i)})x_j}$ for every j.

where

$x:$ input data

$y:$ output data

$n:$ number of data

$\alpha:$ step size

As a data, I have some house sizes in the US and respective house prices.Data can be found here. (Second row is irrelevant data, room number)

My python code is like that:

import numpy as np
import matplotlib.pyplot as plt

file = open("C:/Users/KubilayCan/Desktop/Python/Ben/ML/house_prices.txt", "r+")

house_size, bedrooms, prices = list(), list(), list()

for idx in file:
temp = idx.split(",")
house_size.append(int(temp[0]))
bedrooms.append(int(temp[1]))
prices.append(int(temp[2]))

x = np.array(house_size)
y = np.array(prices)

# plot of given data

plt.xlabel("House Size")
plt.ylabel("Prices")
plt.title("Graph 1:House Prices vs Prices")
plt.scatter(x,y,c="r")
plt.show()

#initialize x and theta

ones = [1] * len(house_size)

x_temp = list(zip(ones, house_size))
x = np.array(x_temp)
x = np.transpose(x)

theta = np.array([0 , 0])

alpha = np.array((0.007,0.00000002))
iteration  = 1000

iter = 0

while iter < iteration:

h = np.matmul(theta,x)

temp = (y-h)
error = np.sum(temp)
theta[0] = theta[0] + alpha[0]*error

temp = (y-h)*x[1]
error = np.sum(temp)
theta[1] = theta[1] + alpha[1]*error
iter +=1

print(theta)

t = np.linspace(0, max(x[1,:]))
lin = theta[1]*t+theta[0]
plt.scatter(x[1,:],y,c="r")
plt.plot(t,lin,c="c")


True result (when closed form of gradient descent is used):

[70610.61964034   134.35956941]


Although I get somehow close, I couldn't get exactly this result. When I play with alpha values my results are dramatically change. How can I find the true values for the alpha variable? Is that only intuition? Or is there any mistake in my python code?

• My answer here is directly responsive to your use-case here because you are optimizing a quadratic objective function using gradient descent. stats.stackexchange.com/questions/145323/…
– Sycorax
Aug 22 '18 at 15:29
• Moreover, you can verify how well your GD method is doing by comparing the coefficient estimates achieved using GD to the coefficients obtained by a standard implementation regression implementation.
– Sycorax
Aug 22 '18 at 15:44