0
$\begingroup$

I am a beginner, and I am trying to use the Lasso to do some regression. I am looking specifically at the LassoLars module in sklearn. What I am really after is recovering the parameter weight vector that is represented by model.coef_. The trouble I am seeing that the weight vector always comes back as a zero vector. Since this is an optimization problem without constraints I figured it should at least return some weights in the parameter for some test data that I'm using.

The objective function that I am tyring to model from the literature looks like this: enter image description here

So i have assumed the the y vector in the Lasso objective function is represented by the rho * vector of ones ( where rho is the mean ). Thats why I have y the way it is as all the same.

The question I have is, is this the right way to use the Lasso to minimize this objective function?

I put together this short example to show my issue. Thanks

import numpy as np
X = np.array( [[ 5.98976150e-03, -1.20984151e-02, -1.22465812e-02, 3.18018512e-03,
  -1.75622040e-03, -1.42396473e-03, 1.50766424e-03, -3.35345720e-03,
   5.19307642e-03],
 [-3.37106026e-03, -6.89156129e-05, 4.81265968e-03, 5.60838622e-03,
   1.22018867e-02, -1.52073535e-02, -1.75285697e-02, -1.51518050e-02,
   2.40028354e-03],
 [-4.05015849e-03, 6.54103374e-03, -4.81555664e-03, 1.72384072e-04,
   5.54351348e-03, -7.61408621e-03, 3.45393320e-04, -1.55521027e-03,
   2.84937377e-03],
 [ 6.82272529e-03, -3.96931710e-03,  1.42365666e-03, -1.07870768e-02,
   3.29016538e-03,  5.47946576e-03, -1.84030370e-02, -6.61630991e-03,
   1.03000002e-02],
 [-5.59911877e-03, -2.04196233e-04, 2.90586730e-03, -1.16229772e-02,
   1.57885915e-02, -3.50405356e-03, -2.56087364e-02, -3.56488119e-02,
   2.97032595e-03]])
n = np.shape( X )[1]
y = [np.mean(X)] * n

from sklearn.linear_model import LassoLars
from sklearn.preprocessing import StandardScaler

model = LassoLars(alpha = 1)
model.fit(X.T, y )
print( model.coef_ )

output: [0,0,0,0,0]

$\endgroup$
4
$\begingroup$

_coef give you parameter vector (w in the cost function formula). You should look at y vector.

y =  [-0.0025973019854199997, -0.0025973019854199997, -0.0025973019854199997, -0.0025973019854199997, -0.0025973019854199997, -0.0025973019854199997, -0.0025973019854199997, -0.0025973019854199997, -0.0025973019854199997]
np.equal.reduce(y) # True

All the values are the same so you recive zeros vector.

To simplify you can imagine that you have two variable X = np.array([1, 2, 3, 4]) and y = np.array([1, 1, 1, 1]).

lin = LinearRegression()
lin.fit(X.reshape(-1, 1), y)
print(lin.coef_) # array([0.])

Let's plot the data and result

Plot

$\endgroup$
  • $\begingroup$ I am trying to fit a model, and I am pretty sure that the y vector in this model is supposed to be the way you see it. In the text it is actually represented by a constant times a vector of ones. @RobJan $\endgroup$ – jeffery_the_wind Aug 2 '18 at 15:03
3
$\begingroup$

Issue 1

In addition to RobJan's answer, I think there is something unintended in your code:

y = [np.mean(X)] * n

This takes the mean of the whole matrix, and replicates it n times. What you might actually want is:

y = np.mean(X, axis=0)

where you actually get the mean of each column separately.

The fact that y was constant before (replicated X times) means that the optimizer could simply use a constant (the intercept_) , and didn't bother to tune the coef_ according to X.

That's what Lasso does - avoids having unnecessary nonzero parameters.

Issue 2

Trying to debug the model, I saw that model.n_iter was 0; which means it gave up instantly. This led me to find out that alpha - the penalty term for the coefficient norm - was too big.

An alpha of 0.001 produced some nonzero coef_s for me.

$\endgroup$
  • $\begingroup$ thanks for the response. I just tried the code with alpha set at 0.001 and much lower, I got zeros every time. I had already tried adjusting the alpha, I'm thinking you may have changed another part of the code? $\endgroup$ – jeffery_the_wind Aug 2 '18 at 15:00
  • $\begingroup$ using y = [np.mean(X)] * n does allow the algo to get some real weights, but I have updated the question and you can see that I think the y vector is supposed to be a vector will all the same value. $\endgroup$ – jeffery_the_wind Aug 2 '18 at 15:15
  • $\begingroup$ The intercept comment was great, I set the fit_intercept param to False, and then I started seeing weights in the weight vector. $\endgroup$ – jeffery_the_wind Aug 2 '18 at 15:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.