# Linear Regression with Lasso Regularization by using scikitlearn and scipy.optimize

i am trying to apply lasso linear regression with both scikitlearn and scipy.optimize min method. However, i cannot reach same result. Code that i created with scipy.optimize can't shrink redundant beta coefficients to zero.

For example let's take alpha parameter as 10,

import pandas as pd
import numpy as np
from matplotlib import pyplot
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from scipy.optimize import minimize
from sklearn import linear_model

df = pd.DataFrame(
[[242, 23.2, 25.4],
[290.0,24.0,26.3],
[340.0,23.9,26.5],
[363.0,26.3,29.0],
[430.0,26.5,29.0],
[450.0,26.8,29.7],
[500.0,26.8,29.7],
],
columns=['y', 'x1', 'x2'])
df

xm=df.iloc[:,1:3]
ym=df.iloc[:,0]

model = linear_model.Lasso(alpha=10)
model.fit(xm, ym)
prediction = model.predict(xm)



then coefficients:

model.coef_


array([ 0. , 43.50726744])

incercept:

model.intercept_


-842.145930232558

then i try to same lasso linear regression with alpha 10 by creating my lasso objective function:


a1=xm.iloc[:,0]
a2=xm.iloc[:,1]

def calc_y(x):
intercept, beta1,beta2 = x
y_predict = intercept + beta1*a1 + beta2*a2
return y_predict

def objective(x):
return np.sum((ym-calc_y(x))**2) + 10*np.sum(abs(x[1:3]))

x0 = np.zeros(3)
no_bnds = (-1.0e10, 1.0e10)
bnds = (no_bnds, no_bnds, no_bnds)

solution = minimize(objective,x0,bounds=bnds)
x = solution.x

print('Solution')
print('intercept = ' + str(x[0]))
print('beta1 = ' + str(x[1]))
print('beta2 = ' + str(x[2]))



Result:

Solution intercept = -917.7266763248599 beta1 = -15.885104892480685 beta2 = 60.62726097294385

What point do you think I'm missing?

Thank you!

• 0. Welcome to CV.SE. 1. You have done almost all the ground work (+1) but you missed a little point! Please see my post below for the necessary change! Commented May 1, 2022 at 16:15

The solver in Lasso is coordinate descent (CG), the default solver in optimize used here is L-BFGS-B; therefore some "little discrepancy" can be there because the two methods are not guaranteed to converge to the same solution. That said, the "big discrepancy" seen is because we are not optimising the same fit. In Lasso the cost function used is $$\frac{1}{2n}\sum (y-\hat{y})^2 +\alpha |\hat{\beta}|$$ (see "Regularization Paths for Generalized Linear Models via Coordinate Descent" from Friedman et al. for more details). In the LASSO implementation shown the cost function is: $$\sum (y-\hat{y})^2 +\alpha |\hat{\beta}|$$; i.e. we want something like (1/(2* len(ym)))*np.sum((ym-calc_y(x))**2) for the first term. (Using that $$\frac{1}{2N}$$, instead of $$\frac{1}{N}$$ is mostly a computational trick, that factor $$2$$ in the denominator greatly simplifies the use of derivatives)
After this amendment the two fits are very close. They aren't exactly the same because CG seems to hit a slightly better minimum when starting from [0, 0, 0] and we used bounds. If we drop the bounds, the optimize solution by L-BFGS-B is even closer to the one from Lasso.