7
$\begingroup$

I'm using Lasso Regularization to avoid overfitting & multicollinearity between two features (X1 and X2), nowing that I have 14 independent features. I got some good results for some features, Lasso was able to reduce the coefficient to 0, but for other features the linear regression coefficient was less than Lasso (same thing for Ridge).

lr = LinearRegression()
lr.fit(X, Y)
lr_coeff = lr.coef_
lr_intercept = lr.intercept_

lasso = Lasso(alpha=10)
lasso.fit(X, Y)
lasso_coeff = lasso.coef_
lasso_intercept = lasso.intercept_

Result:

    lr_coeff  lr_intercept  lasso_coeff  lasso_intercept
0   0.968567      16.01858     0.000000       103.471224
1   1.743420      16.01858     1.730920       103.471224
2   5.221518      16.01858     3.931450       103.471224
3   4.769328      16.01858     3.186003       103.471224
4   6.341612      16.01858     4.265931       103.471224
5   2.272504      16.01858     1.277541       103.471224
6   3.104016      16.01858     1.648253       103.471224
7   1.418943      16.01858     0.667189       103.471224
8   1.144834      16.01858     0.000000       103.471224
9   0.138457      16.01858     0.000000       103.471224
10  1.272995      16.01858     0.693323       103.471224
11  0.188450      16.01858     0.503958       103.471224
12 -2.334245      16.01858    -0.167953       103.471224
13 -0.475823      16.01858     0.124608       103.471224
14  0.489548      16.01858     0.512034       103.471224

Sincerely,

$\endgroup$
2
  • 3
    $\begingroup$ To be honest I think it is perfectly fine. $\endgroup$ Commented Feb 14, 2022 at 10:38
  • $\begingroup$ What is the question? $\endgroup$
    – Firebug
    Commented Feb 14, 2022 at 16:32

2 Answers 2

13
$\begingroup$

As German Demidov notes, this is perfectly fine. The Lasso will shrink some of your coefficients to zero, but it does not have the property of shrinking all coefficients compared to the OLS estimate. Rather, it may increase some coefficients to "compensate" for the ones it has shrunk. There is nothing to worry about. (A very good question, though.)

$\endgroup$
8
$\begingroup$

Lasso coefficients can shrink again while you get closer to the OLS solution.

See for instance: Why under joint least squares direction is it possible for some coefficients to decrease in LARS regression?

Here is an image of the relationship between the coefficients and the error. The lasso balances the error (depicted by the green surface) and the size of the coefficients (the red surface). For a given amount of regularisation it might be that some some parameters 'overshoot' and are larger than the actual OLS. By having these parameters larger you will have other parameters lower.

intuitive view of lasso path

This situation happens when one parameter can take the role of several others. In that case, initially this parameter will be able to model the outcome very well even with a small coefficient (and one that is above the true model coefficient), but if you allow the total of coefficients to be larger then the others may catch up.

A clear illustration of this principle is in this question where a coefficient that should be zero is initially possitive. This happens because the parameter models the outcome better than the true model when the penalty is high: Is Ridge more robust than Lasso on feature selection?

$\endgroup$
1
  • $\begingroup$ Thank you, this is very helpful! $\endgroup$ Commented Feb 14, 2022 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.