I was reading ISLR and implemented the least squares approach for a linear regression model on the autos data set which comes with the book. For the least squares approach I used only one predictor(acceleration) .

After that I tried to fit the model with two predictors (acceleration, vehicle weight). I used a quadratic equation of the form $\hat y = m_1x^2 + m_2x + b$

Then I ran gradient descent with following parameters:-
* Learning rate = [0.01, 0.001, 0.0001]
* epochs = 1000

Dataset shape:-
train inputs: (296, 2)
train outputs: (296, 1)
test inputs: (99, 2)
test outputs: (99, 1)


This is the data(shown above)

def y_hat_quad(w1,w2,b0,x):
    """ returns the result of the quadratic equation y = w1.x^2 + w2.x + b0

    w1, w2 are slopes
    b0 is the intercept
    x is the input data


    return np.dot(np.square(x), w1) + np.dot(x, w2) + b0

# weight initialisation
m1 = np.zeros((2, 1),dtype=np.float32)
m2 = np.zeros((2, 1),dtype=np.float32)
b  = np.zeros((1, 1),dtype=np.float32)
# I've tried sampling from normal distribution as well

### length of training set
m_tr = len(train_input)

for i in range(epochs):
    ### calculate y_hat for entire dataset
    y_quad = y_hat_quad(m1,m2,b,train_input)

    ### run gradient descent optimizer
    ### d(mse)/dm1 = d(mse)/dy_hat * d(y_hat)/d(m1)
    d_mse = y_quad-train_output

    d_mse_dm1 = np.dot(d_mse.T, np.square(train_input))

    ### d(mse)/dm1 = d(mse)/dy_hat * d(y_hat)/d(m2)
    d_mse_dm2 = np.dot(d_mse.T, train_input)

    ### d(mse)/db = d(mse)/dy_hat * d(y_hat)/d(b)
    d_mse_db = np.sum(d_mse)

    ### updating slope and intercept
    m1 = m1 - (lr/m_tr) * d_mse_dm1.T
    m2 = m2 - (lr/m_tr) * d_mse_dm2.T
    b = b - (lr/m_tr) * d_mse_db

Following it the code I have used to plot the fit

# accuracy calculation
plt.plot(train_input[:,0],train_output, 'xg')
plt.plot(train_input[:,1],train_output, 'xr')
plt.plot(train_input[:,0],y_quad, '-b')
plt.plot(train_input[:,1],y_quad, '-b')

Which gives the following result

fitted model

Following is the training loss decay

training loss

How do I fix this? Thanks.

Edit 1: as suggested by usεr11852, I've sorted the result but it fits poorly, how can I improve the quality of the fit?

training set

test set

  • $\begingroup$ Please sort your prediction points by the order of the x axis and replot. $\endgroup$ – usεr11852 May 29 at 8:12
  • $\begingroup$ Thanks for the reply, I have sorted the result. But it isn't a good fit. What can I do about it? Thanks. $\endgroup$ – quantumbiker May 29 at 9:23
  • $\begingroup$ It looks pretty expected to me, what did you expect exactly? And why you have these steps? These look pretty odd... (I will look it up tomorrow night properly super busy now) $\endgroup$ – usεr11852 May 29 at 9:45
  • $\begingroup$ The dataset appears to have a curve, it's not in a straight line so I thought a quadratic equation would account for it. But the fitted curve is actually somewhat a straight line. I wanted it to simulate the curve. Btw thanks for your time. $\endgroup$ – quantumbiker May 29 at 9:49
  • $\begingroup$ Did you try using a general function/package instead of your own code and compare the results? The fitted jumps do indeed look suspicious. Or is checking your implementation for errors part of the question? $\endgroup$ – Nutle May 29 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.