The well-known closed-form solution of Ridge regression is:
I am trying to implement the closed-form using NumPy and then compare it with sklearn. I can get the same result when there is no fit_intercept (fit_intercept = False). However, when fit_intercept = True, I cannot get the same results even though I have tried several sklearn Ridge solvers. To implement the above formula with NumPy when intercept is not 0, I concatenated 1 to all feature vectors. Below is my code:
#data set (X,y)
#using formular
N,M = X.shape
one = np.ones((N, 1))
Xbar = np.concatenate((one, X), axis = 1) #concatenate 1 to all features vectors
I = np.identity(M+1)
XT = Xbar.T
XTX = XT.dot(Xbar)
INV = np.linalg.inv(XTX+alpha*I)
beta = INV.dot(XT.dot(y))
print('beta', beta)
and to calculate beta by using sklearn, I implemented the following code:
clf = Ridge(alpha,fit_intercept=True)
clf.fit(X, y)
print(clf.intercept_, clf.coef_)
However, after trying several solvers of Ridge, I still obtain very different values for weight vectors by the 2 codes. What did I do wrong here?