Significant Difference in prediction when using library and coding from scratch in Multiple Linear Regression [duplicate]

Question

I have been trying to implement multiple linear regression from scratch after implementing it using sklearn. The values predicted using sklearn is very accurate whereas the values predicted by the code which I have coded from scratch is not so much accurate. This is the dataset that I am using (I have provided the entire dataset in my Github link mentioned below along with the source code of my implementation)

I am predicting profit of an organisation by seeing the 4 independent variables. Notice that we also have a categorical value here, on which I have done 1 hot encoding, and I have also scaled all the data.

I am using gradient descent to reach the optimal values for slope for the independent variables. The function I have written for gradient descent updation is as follows.

#building the model

#lets start with a random value of m and c

m_1=0
m_2=0
m_3=0
m_4=0
m_5=0
m_6=0

c=0

l=0.0001 #l is the learning rate



n=float(len(X_train)) # n=number of training data, we are converting it to float since we will need to divide n

mse=[]  #we are creating a list for storing mean square errors for each iteration so that we can plot it

for i in range (0,20000):
    Y_pred=m_1*X_train[:,0:1]+m_2*X_train[:,1:2]+m_3*X_train[:,2:3]+m_4*X_train[:,3:4]+m_5*X_train[:,4:5]+m_6*X_train[:,5:6]+c
    
    mse.append(numpy.sum((Y_pred-Y_train)**2)/n)
    D_m_1= (2/n) * sum(X_train[:,0:1]*(Y_pred-Y_train))
    D_m_2= (2/n) * sum(X_train[:,1:2]*(Y_pred-Y_train))
    D_m_3= (2/n) * sum(X_train[:,2:3]*(Y_pred-Y_train))
    D_m_4= (2/n) * sum(X_train[:,3:4]*(Y_pred-Y_train))
    D_m_5= (2/n) * sum(X_train[:,4:5]*(Y_pred-Y_train))
    D_m_6= (2/n) * sum(X_train[:,5:6]*(Y_pred-Y_train))
    D_c= (2/n) * sum(Y_pred-Y_train)
    
    #Andrew Ng has not taken 2/n but instead 1/n as his cost function is multiplied by 1/2 to reduce complexity
    
    m_1=m_1-l*D_m_1
    m_2=m_2-l*D_m_2
    m_3=m_3-l*D_m_3
    m_4=m_4-l*D_m_4
    m_5=m_5-l*D_m_5
    m_6=m_6-l*D_m_6
    c=c-l*D_c
   
 
    
plt.plot(mse)

#see the plot below, we can see the point where MSE stabilises. Run the above loop till that point and take MSE

This is the function that I have written, and in the other code where I have used sklearn library the code looks like this,

#now we will fit the linear regression model

from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt 
regressor= LinearRegression() 
regressor.fit(X_train,Y_train)

#predicting the test results
Y_pred=regressor.predict(X_test)

But there is a significant difference between the predicted value of the 2 code and as obvious the sklearn is producing good results. I am not getting why the code which I have written from scratch is having low accuracy. I am providing the dataset as well as the code (where I have written the gradient descent) below.

Link to code and dataset--> https://github.com/Hirak999/StackExchange-Doubts

Answer 1

I am using gradient descent to reach the optimal values for slope for the independent variables.

Here is the answer. (Stochastic) gradient descent is not the only way to optimize linear regression model. There is an analytic solution $-$ the parameters $\mathbf{w}$ that minimize squared error loss are determined by a closed-form, ordinary least squares (OLS) solution $$ \mathbf{w} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} $$ sklearn.linear_model.LinearRegression is an OLS estimator, and it turns out that your data is fit better with this model.