I took a method to determine the best features for a classifier from somewhere and I wanted to ask if it is reliable or not:

I was trying to, very simply, select the best features for a Logistic Regression classifier.

Using Lasso().coef_ it turned out that everytime I switched the training and testing data indexs (with a random ShuffleSplit, so in a CrossValidation process), the 0 lasso coefs changed. So, to visualize it:

Lasso_coefs_obtained = [0,   0.9, 0,  0,   0.1]

Lasso_coefs_obtained = [0.3, 0,   0,  0.7, 0  ]

Lasso_coefs_obtained = [0.1, 0.8, 0,  0.2, 0.2]

In order to determine if a feature coefficient was really null, I would sum the three coefficients lists and determine the 0 elements (which must be those elements that are 0 in the three lists).

Sum_coefs = [0.4, 1.7, 0,  0.9, 0.3]

In my case, I did this method with 10 loops (I have 10 coefficients lists over 72 variables). After 10 loops, only 55 coefficients remain non-zero. So, I delated 17 features for the classifier.

Is this method reliable or I'm just making up an algorithm with no guarantees?


1 Answer 1


Clarifying your approach

the 0 lasso coefs changed

What value of $\lambda$ (the Lasso parameter) did you use and how did you determine it ? Your approach seems confusing:

  • Have you performed k-fold cross validation on your training set for a range of $\lambda$ values and chosen the $\lambda$ that gives the best results ?
  • I would suggest starting with cross validation without replacement so as to avoid a kind of bootstrapping effect (i.e. careful with the shufflesplit)

A good introduction to combining Lasso and cross validation is provided by the inventor of the Lasso, Robert Tibshirani, pages 15 and 16 here. Also here: Lasso cross validation and here

enter image description here

Some reasons for why Lasso coefficients would be different

  • Wrong approach: if you only use the default $\lambda$ value of your algorithm (e.g. from R or Sklearn), mix up the data set with replacement and perform Lasso you are likely going to obtain different parameter values.
  • Too few data points: if your data set is too small, or if $K$ of the K-fold CV is too large, your model will be fitted on a data set which is not representative of the overall data set, which would explain different results across the folds
  • More features than data points: CV and Lasso are unstable in this case
  • Unstable model or extreme collinearity: If you use a standard approach to select features but still get different coefficients, my intuition is that your features are so highly correlated that they cause the algorithm to struggle / encounter numerical issues / become unstable

Some additional sources on the topic:


Provided that none of the above issues apply, bootstrapping your data set and performing standard k-fold CV can still be useful

  • You will need to repeat the boostrap experiment hundreds or thousands of times (not 10)
  • You can perform statistical inference on the bootstrap results such as a bootstrapped confidence intervals, statistical significance tests etc..
  • Careful when interpreting these results however, as bootstrap inference has its own limitations, bias and interpretation issues (a non trivial topic - look on stackexchange)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.