I was reading ISLR and implemented the least squares approach for a linear regression model on the autos data set which comes with the book. For the least squares approach I used only one predictor(acceleration) .
After that I tried to fit the model with two predictors (acceleration, vehicle weight). I used a quadratic equation of the form $\hat y = m_1x^2 + m_2x + b$
Then I ran gradient descent with following parameters:-
* Learning rate = [0.01, 0.001, 0.0001]
* epochs = 1000
train inputs: (296, 2)
train outputs: (296, 1)
test inputs: (99, 2)
test outputs: (99, 1)
This is the data(shown above)
def y_hat_quad(w1,w2,b0,x): """ returns the result of the quadratic equation y = w1.x^2 + w2.x + b0 w1, w2 are slopes b0 is the intercept x is the input data """ return np.dot(np.square(x), w1) + np.dot(x, w2) + b0 # weight initialisation m1 = np.zeros((2, 1),dtype=np.float32) m2 = np.zeros((2, 1),dtype=np.float32) b = np.zeros((1, 1),dtype=np.float32) # I've tried sampling from normal distribution as well ### length of training set m_tr = len(train_input) for i in range(epochs): ### calculate y_hat for entire dataset y_quad = y_hat_quad(m1,m2,b,train_input) ### run gradient descent optimizer ### d(mse)/dm1 = d(mse)/dy_hat * d(y_hat)/d(m1) d_mse = y_quad-train_output d_mse_dm1 = np.dot(d_mse.T, np.square(train_input)) ### d(mse)/dm1 = d(mse)/dy_hat * d(y_hat)/d(m2) d_mse_dm2 = np.dot(d_mse.T, train_input) ### d(mse)/db = d(mse)/dy_hat * d(y_hat)/d(b) d_mse_db = np.sum(d_mse) ### updating slope and intercept m1 = m1 - (lr/m_tr) * d_mse_dm1.T m2 = m2 - (lr/m_tr) * d_mse_dm2.T b = b - (lr/m_tr) * d_mse_db
Following it the code I have used to plot the fit
# accuracy calculation plt.plot(train_input[:,0],train_output, 'xg') plt.plot(train_input[:,1],train_output, 'xr') plt.plot(train_input[:,0],y_quad, '-b') plt.plot(train_input[:,1],y_quad, '-b') plt.show()
Which gives the following result
Following is the training loss decay
How do I fix this? Thanks.
Edit 1: as suggested by usεr11852, I've sorted the result but it fits poorly, how can I improve the quality of the fit?