I'm practicing machine learning in python and trying to implement batch gradient descent algorithm by myself. Mathematically algorith is defined as follows:

$\theta_j = \theta_j + \alpha \sum_{i=0}^{n}{(y^{(i)}-h_{\theta}^{(i)})x_j}$ for every j.


$x:$ input data

$y:$ output data

$n:$ number of data

$\alpha: $ step size

As a data, I have some house sizes in the US and respective house prices.Data can be found here. (Second row is irrelevant data, room number)

My python code is like that:

import numpy as np
import matplotlib.pyplot as plt

file = open("C:/Users/KubilayCan/Desktop/Python/Ben/ML/house_prices.txt", "r+")

house_size, bedrooms, prices = list(), list(), list()

for idx in file:
    temp = idx.split(",")

x = np.array(house_size)
y = np.array(prices)

# plot of given data

plt.xlabel("House Size")
plt.title("Graph 1:House Prices vs Prices")

# batch gradient algorith

#initialize x and theta

ones = [1] * len(house_size)

x_temp = list(zip(ones, house_size))
x = np.array(x_temp)
x = np.transpose(x)

theta = np.array([0 , 0])

alpha = np.array((0.007,0.00000002))
iteration  = 1000

iter = 0

while iter < iteration: 

    h = np.matmul(theta,x)

    temp = (y-h)
    error = np.sum(temp)
    theta[0] = theta[0] + alpha[0]*error

    temp = (y-h)*x[1]
    error = np.sum(temp)
    theta[1] = theta[1] + alpha[1]*error
    iter +=1


t = np.linspace(0, max(x[1,:]))
lin = theta[1]*t+theta[0]
plt.title("Graph 2: Batch Gradient Descent")

True result (when closed form of gradient descent is used):

[70610.61964034   134.35956941]

enter image description here

Although I get somehow close, I couldn't get exactly this result. When I play with alpha values my results are dramatically change. How can I find the true values for the alpha variable? Is that only intuition? Or is there any mistake in my python code?

  • $\begingroup$ My answer here is directly responsive to your use-case here because you are optimizing a quadratic objective function using gradient descent. stats.stackexchange.com/questions/145323/… $\endgroup$ – Sycorax Aug 22 '18 at 15:29
  • $\begingroup$ Moreover, you can verify how well your GD method is doing by comparing the coefficient estimates achieved using GD to the coefficients obtained by a standard implementation regression implementation. $\endgroup$ – Sycorax Aug 22 '18 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.