Linear Regression with Lasso Regularization by using scikitlearn and scipy.optimize i am trying to apply lasso linear regression with both scikitlearn and scipy.optimize min method.
However, i cannot reach same result. Code that i created with scipy.optimize can't shrink redundant beta coefficients to zero.
For example let's take alpha parameter as 10,
import pandas as pd
import numpy as np
from matplotlib import pyplot
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from scipy.optimize import minimize
from sklearn import linear_model

df = pd.DataFrame(
[[242, 23.2, 25.4],
[290.0,24.0,26.3],
[340.0,23.9,26.5],
[363.0,26.3,29.0],
[430.0,26.5,29.0],
[450.0,26.8,29.7],
[500.0,26.8,29.7],
],
columns=['y', 'x1', 'x2'])
df


xm=df.iloc[:,1:3]
ym=df.iloc[:,0]

model = linear_model.Lasso(alpha=10)
model.fit(xm, ym)
prediction = model.predict(xm)


then coefficients:
model.coef_

array([ 0.        , 43.50726744])
incercept:
model.intercept_

-842.145930232558
then i try to same lasso linear regression with alpha 10 by creating my lasso objective function:

a1=xm.iloc[:,0]
a2=xm.iloc[:,1]

def calc_y(x):
    intercept, beta1,beta2 = x
    y_predict = intercept + beta1*a1 + beta2*a2
    return y_predict

def objective(x):
    return np.sum((ym-calc_y(x))**2) + 10*np.sum(abs(x[1:3]))


x0 = np.zeros(3)
no_bnds = (-1.0e10, 1.0e10)
bnds = (no_bnds, no_bnds, no_bnds)

solution = minimize(objective,x0,bounds=bnds)
x = solution.x

print('Solution')
print('intercept = ' + str(x[0]))
print('beta1 = ' + str(x[1]))
print('beta2 = ' + str(x[2]))


Result:
Solution
intercept = -917.7266763248599
beta1 = -15.885104892480685
beta2 = 60.62726097294385
What point do you think I'm missing?
Thank you!
 A: The solver in Lasso is coordinate descent (CG), the default solver in optimize used here is L-BFGS-B; therefore some "little discrepancy" can be there because the two methods are not guaranteed to converge to the same solution. That said, the "big discrepancy" seen is because we are not optimising the same fit. In Lasso the cost function used is  $\frac{1}{2n}\sum (y-\hat{y})^2 +\alpha |\hat{\beta}|$ (see "Regularization Paths for Generalized Linear Models
via Coordinate Descent" from Friedman et al. for more details). In the LASSO implementation shown the cost function is: $\sum (y-\hat{y})^2 +\alpha |\hat{\beta}|$; i.e. we want something like (1/(2* len(ym)))*np.sum((ym-calc_y(x))**2) for the first term. (Using that $\frac{1}{2N}$, instead of $\frac{1}{N}$ is mostly a computational trick, that factor $2$ in the denominator greatly simplifies the use of derivatives)
After this amendment the two fits are very close. They aren't exactly the same because CG seems to hit a slightly better minimum when starting from [0, 0, 0] and we used bounds. If we drop the bounds, the optimize solution by L-BFGS-B is even closer to the one from Lasso.
