This is not an Homework assignment. I am trying to implement the gradient descend explained in the link (Page 5) [http://cs229.stanford.edu/notes/cs229-notes1.pdf].I am able to successfully run the code and getting optimal coefficients suggested by closed form solution of simple regression but the problem is that it's taking 400000 iterations to achieve the desired solution. And if i increase my learning rate from 0.00003 to 0.0003, it becomes an gradient ascend algorithm i.e. loss value explodes to infinity. How do i reduce these large number of iterations and still get the closed form solution ? And how one should set learning rate for their gradient descend algorithm ? Any good useful advice would be much appreciated.
import pandas as pd import numpy as np # Data processing code advertising_data = pd.read_csv("http://www-bcf.usc.edu/~gareth/ISL/Advertising.csv", index_col=0) target = np.array(advertising_data.Sales.values) advertising_data["ones"] = np.ones(200) advertising_data = advertising_data[["ones", "TV"]] features = np.array(advertising_data.values) # Gradient Descend implementation here def error_ols(target, features): def h(betas): error = target - np.dot(features, betas) return error return h def ols_loss(errors): return np.sum(errors*errors) def gradient_descend(initial_guess, learning_step, gradient, iterations = 400000): for i in range(0, iterations): update = initial_guess + learning_step*gradient( initial_guess) initial_guess = update error = error_ols(target, features)(update) print ols_loss(error) return update def ols_gradient(target, features): def h(betas): error = target - np.dot(features, betas) return np.dot(error, features)/200 return h gradient_function = ols_gradient(target, features) initial_guess = np.array([1,1]) print gradient_descend(initial_guess, 0.00007, gradient_function)