3
$\begingroup$

I have a data set with n=199 and p=130. I need to reduce the number of predictors for a regression. I did a Lasso regression but the numer of optimal variables changes dramatically (range from 6 to 41 predictors recommended by Lasso) when I change the seed number input. How could I determine the optimal set of variables to include in the model?

These are my different number of variables that i got.

variables
  [1] 24 21 29 24 21 21 24 33 24 33 24 39 21 21 24 21 29 21 29 29 24 29 29 29 24 18 29 24 21 39 39 21 21 29 33 39 24 41
 [39] 24 13 29 13 29 29 21 24 24 39 24 33 24 29 33 45 21 21 21 24 29 33 29 29 18 21 21 24 21 21 33 24 33 21 21 21 21 29
 [77] 41 24 39 21 21 24 21 39 29 21 39 21 41 21 33 21 33 33 21 33 21 24 21 29 21 21 21 29 24 21 21 21 29 29 24 21 24 21
[115] 21 21 21 33 18 21 39 29 29 21 42 21 13 24 33 21 39 24 29 33 21 41 29 24 42 33 41 21 21 21 13 24 21 24 24 39 24 41
[153] 50 33 21 24 24 18 29 24 39 21 21 24 33 42 21 21 24 29 24 21 24 24 24  6

My target is a dummy variable and my code is this

variables=rep(0) # vector with the numbers of variables with differente seed number
num.var<-0 #number of variable for any best lambda
iter<-0  #number of iteration
seed_input=500 # first seed number

while(iter <177){
seed_input=seed_input+floor(rnorm(1,10,2)) # change the seed number in diferent iteration
set.seed(seed_input) #use this seed number 
modelo<-glmnet(independent,dependent, family = "binomial", alpha=1) #fitting lasso regression
cv.modelo <- cv.glmnet(independent, dependent, alpha=1) 
best.lambda <- cv.modelo$lambda.min #saving best lambda #saving best lambda

    # Creating a vector with number of variables
n=dim(a)[1] 
    vars=rep(0)
    j=1
    for(i in 2:n){ 
      if(a[i,1]!=0){vars[j]=i
      j=j+1}
    }
    num.var[iter]<-length(vars) #save the numbers of variables in a vector
    iter=iter+1
    }

I hope that somebody can help me. Thanks.

| cite | improve this question | | | | |
$\endgroup$
  • $\begingroup$ Please type your question as text, do not just post a photograph (see here). $\endgroup$ – gung - Reinstate Monica Feb 1 '17 at 1:05
  • $\begingroup$ I suggest putting the whole executable example. How did you get those numbers of variables? $\endgroup$ – Pere Feb 1 '17 at 1:09
  • $\begingroup$ Lasso is typically used to reduce the feature space of the model (ie number of attributes, number of data columns), not the number of samples. From the question, it is difficult to discern what is n (typically designates sample size), what is p and what you're trying to reduce. $\endgroup$ – Gene Burinsky Feb 1 '17 at 1:09
  • $\begingroup$ Cross-validation will tell you which set of variables yields the "best accuracy" or best out of sample prediction. $\endgroup$ – Gene Burinsky Feb 1 '17 at 1:10
  • $\begingroup$ Thanks to all for your awnser. @GeneBurinsky, n is the size of my sample and p is the number of variables. I need to reduce the number of variables but i don't understand why this number of variables changes with differents set seed. $\endgroup$ – Frank Josué Castillo Isisola Feb 1 '17 at 1:22
4
$\begingroup$

The reason your results change is because of the randomization used to cross-validate the estimates. In k-fold cross-validation, (ie what I think glmnet uses), each time cross-validation is run, a part of your data is randomly selected to be a training set and the remainder of the data is selected to be the test set. When you set different seeds, you are changing which parts of the data end up in the training and testing parts. As a result, at each different seed, glmnet fits the models to a different training set and cross-validation tests it on a different testing set. Consequently, the variables selected at each seed may vary.

Dirty ways around this: you could leave the seed alone or fit a large number of times on different seeds seeds and select the variable set that is chosen most often.

As mentioned by Gammer in the comments, leave-one-out cross-validation will not suffer the drawback from seed changes but may take a lot longer to run.

Check this cv question, the OP had a similar concern and the response may help answer your question of "how" do I select the best set.

| cite | improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Thanks a lot. Do you think that It happend because i have a small data set or it doesn't have influence? $\endgroup$ – Frank Josué Castillo Isisola Feb 1 '17 at 2:10
  • $\begingroup$ Your data are quite small and that may have something to do with it though distinct clustering within the data regardless of size could also influence this. I'm sure there are numerous other reasons that could cause this. It's difficult to tell as I'm not familiar with your data $\endgroup$ – Gene Burinsky Feb 1 '17 at 2:16
  • 1
    $\begingroup$ Yes, the random determination of folds is the reason for the inconsistent results, but note that this shouldn't affect leave-one-out cross validation, because the folds are deterministic in that case. $\endgroup$ – gammer Feb 1 '17 at 3:28
  • $\begingroup$ @FrankJosuéCastilloIsisola, Frank Harrell had some related answers here on Cross Validated (about what to do if LASSO is unstable), you may check them. $\endgroup$ – Richard Hardy Feb 1 '17 at 6:28
  • $\begingroup$ @RichardHardy, could you give me the link ? Thank you! $\endgroup$ – Frank Josué Castillo Isisola Feb 1 '17 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.