The well-known closed-form solution of Ridge regression is: enter image description here

I am trying to implement the closed-form using NumPy and then compare it with sklearn. I can get the same result when there is no fit_intercept (fit_intercept = False). However, when fit_intercept = True, I cannot get the same results even though I have tried several sklearn Ridge solvers. To implement the above formula with NumPy when intercept is not 0, I concatenated 1 to all feature vectors. Below is my code:

#data set (X,y)
#using formular
N,M = X.shape

one = np.ones((N, 1))
Xbar = np.concatenate((one, X), axis = 1)  #concatenate 1 to all features vectors
I = np.identity(M+1)
XT = Xbar.T
XTX = XT.dot(Xbar)
INV = np.linalg.inv(XTX+alpha*I)
beta = INV.dot(XT.dot(y))
print('beta', beta)

and to calculate beta by using sklearn, I implemented the following code:

clf = Ridge(alpha,fit_intercept=True)
clf.fit(X, y)
print(clf.intercept_, clf.coef_)

However, after trying several solvers of Ridge, I still obtain very different values for weight vectors by the 2 codes. What did I do wrong here?


I found the answer here: Understanding Ridge Linear Regression in sci-kit learn

To summarize, sklearn ridge regression does not add penalty to the intercept term as the analytical formula does.

  • $\begingroup$ Same for R using the glmnet package. $\endgroup$ – chl Oct 13 '20 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.