7
$\begingroup$

I'm using from sklearn.linear_model import Lasso in Python 2.7.6

I wrote a script that I've used for doing a Lasso regression for my Features (X) and my Targets (y) . I've used it before and it works, I'm using it on a new dataset (completely different type of data) and I'm getting all 0 coefficients.

What does this mean? Is there anything I can change or tweak to get data?

I've tried different alpha parameters. Here's my function below. I use this class system to store my models and stuff. Let me know if it is confusing or needs to be generalized. I think it's pretty straight forward. My notation is DF_ = DataFrame, D_ = Dictionary, SR_ = Series

#Create the models for store them
from sklearn.cross_validation import LeavePOut
from sklearn.linear_model import Lasso
import time
from collections import defaultdict

class Models:
    def __init__(self,target=None,description=None,models=[],duration=0.0):
        self.target = target; self.models = models; self.duration = duration; self.description = description
    def summation(self):
        return(float(sum([q[1] for q in self.models])))
    def score(self):
        return(self.summation()/len(self.models)) 

def synthesis(description, query_targets, D_target_Models, DF_attributes, DF_targets,alpha = 1):
    """
    Updates Model object
    Parameters:
    [description] key for dictionary of D_target_Models that stores instances of class
    [query_targets] list of targets to make models for in DF_targets
    [D_target_Models] dictionary of dictionaries:
        Outer dict: {description:targets}; 
        Inner dict: {target:model_instance}
    [DF_attributes] Pandas DataFrame of attributes (index = sample, column = attribute)
    [DF_targets] Pandas DataFrame of targets (index = sample, column = targets)
    [alpha] lambda for regression method
    """
    lpo = LeavePOut(len(DF_attributes.index)/1000, p=2)
    #Check order of indices
    if (list(DF_attributes.index) == list(DF_targets.index)) == True:
#         X.index = Y.index = range(len(X.index))
        for target in query_targets:
            #Create target instance
            D_target_Models[description][target] = Models(target=target)
            #Get query column for target
            SR_target = DF_targets[target]

            #Create and train models
            models = []
            for train_indices,test_indices in lpo:
                #Check if all test sets have values
                #NOTE!(These conditionsaren't essential for understanding the script.  It's how I ensured there were no NAs)
                condition_1 = all([(pd.isnull(SR_target.iloc[test_i]) == False) for test_i in test_indices])
                condition_2 = DF_attributes.iloc[test_i].isnull().values.any() == False
                condition_3 = None #Impute missing data on DF_attributes
                conditions = [condition_1,condition_2]

                if all(conditions) == True: #Assumes data is present for all features
                    #Create model
                    duration_start = time.time() #So I can time the modeling, not essential
                    model = Lasso(alpha=alpha)

                    #Update training indices with non-null target/sensitivity indices
                    train_indices = [train_i for train_i in train_indices if pd.isnull(SR_target.iloc[train_i]) == False]
                    #Assign X and y
                    train_X = DF_attributes.iloc[train_indices,:]
                    test_X = DF_attributes.iloc[test_indices,:]
                    train_y = SR_target.iloc[train_indices]
                    test_y = SR_target.iloc[test_indices]
                    #Fit model
                    model.fit(train_X,train_y)

                    #Predict
                    predicted_values = model.predict(test_X)
                    correct_values = test_y
                    accuracy = int((predicted_values[0] > predicted_values[1]) == (correct_values[0] > correct_values[1]))
                    if accuracy == 1:
                        if len(set(model.coef_)) > 1:
                            print(set(model.coef_)) #ALL COEFFICIENTS ARE 0.0
                    #Store models
                    models.append((model,accuracy,test_indices))

                #Store time for models
            D_target_Models[description][target].models = models
            D_target_Models[description][target].duration = float(time.time() - duration_start)
        return(D_target_Models)
    else:
        return("DF_attributes.index != DF_target.index")
$\endgroup$
  • $\begingroup$ It it possible that your variables simply aren't strongly related to the response? (Note that if you want someone to read through your code & look for problems, that would be off topic here--you could try Code Review.) $\endgroup$ – gung Oct 28 '15 at 22:26
  • $\begingroup$ Is that what a 0 coefficient would mean for all the attributes? I ran it with LassoCV instead of Lasso and got coefficients. So does that mean that my alpha was the problem? $\endgroup$ – O.rka Oct 28 '15 at 23:02
  • $\begingroup$ That could have been it. $\endgroup$ – gung Oct 28 '15 at 23:03
12
$\begingroup$

Here, the key fact about LASSO regression is that it minimizes sum of squared error, under the constraint that the sum of absolute values of coefficients is less than some constant $c$. (See here.) So, for all of the coefficients to be zero, there must be no vector of coefficients with summed absolute value less than $c$ that improves error.

For another view, consider the LASSO loss function:

$$\sum_{i = 1}^n (Y_i - X_i^T\beta) + \lambda\sum_{j=1}^p|\beta_j|$$

As put in the tutorial referenced above, "If $\lambda$ is sufficiently large, some of the coefficients are driven to zero, leading to a sparse model." For it to be the case that zero coefficients minimize this function, $\lambda$ must be large enough that any improvement in error (the left term) is less than the added loss from the increased norm (the right term).

It's common to use cross validation to set this parameter such that the model minimizes CV error. This could be why LassoCV gave you different results—it may have set $\lambda$ for you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.