4
$\begingroup$

I used the glmnet-package to do a regression + variable-selection with Lasso. I had n=100 oberservations and p=200 covariables. I always read that after variable-selection with the Lasso there a maximal min(n,p) covariables in the resulting model. But I got a model with 102 covariables. Is there anyone with a explanation for that result? In my opinion there should be maximal 100 variables in the model. Thanks!

$\endgroup$
1
  • 2
    $\begingroup$ Can you show your code and give a sense of your variables? At this point, it isn't clear if this is a problem with your code (in which case it would be off topic), with glmnet (unlikely, but also off topic) or a problem with LASSO on your data (which would be on topic). $\endgroup$ – Peter Flom Jul 25 '15 at 12:57
3
$\begingroup$

Lasso is a linear regression with a small additional constraint: together with prediction errors, large absolute values of regression coefficients are penalized (so called l1 penalty as opposed to l2 in ridge regression). The amount of penalization is governed by regularization parameter lambda. In general, the higher the lambda the fewer coefficients are left in the model. Optimal number of coefficients, or optimal model if you wish, is checked via cross-validation.

Where did you get "maximal min(n,p)"? It does not sound right according to what I learnt in my statistical courses and my experience with lasso, as number of coefficients can be any depending on value of lambda and your data.

As a general side note, lasso models are notorious for instability, so you may wish try elasticnet, which is a mix of lasso and ridge. As usual, use cross-validation to tune parameters.

Hope this helps.

$\endgroup$
3
  • 2
    $\begingroup$ Elastic net is a good suggestion. For either lasso or elastic net a bootstrap study of the volatility of selected features is important to do. $\endgroup$ – Frank Harrell Jul 26 '15 at 11:42
  • $\begingroup$ Thank you for your quick help. I read about min(n,p) in the elastic.net publication by zou and hastie. they write "In the p>n case, the lasso selects at most n variables before it saturates, because of the nature of the convex optimization problem. This seems to be a limiting feature for a variable selection method." So I don`t know how I could get these results $\endgroup$ – Theresa Jul 28 '15 at 8:03
  • 2
    $\begingroup$ Hanna, what is your problem? Why do you need to find p>n features? Why are you not satisfied with p<n features? Have you cross-validated your results? Why are you doing feature selection in the first place? Depending on your answers, you may turn to other regression methods with built-in feature selection, e.g. Random Forest. Before going any further, I would strongly suggest reading this summary of Feature selection in R (via caret package): topepo.github.io/caret/featureselection.html $\endgroup$ – Sergey Bushmanov Jul 31 '15 at 8:34
1
$\begingroup$

In cases like this, I always try another variable selection procedure and see how much they are in agreement.

VIF (https://en.wikipedia.org/wiki/Variance_inflation_factor)

Here is a paper on VIF "VIF Regression: A Fast Regression Algorithm for Large Data" (https://statistics.wharton.upenn.edu/files/?whdmsaction=public:main.file&fileID=3486) (Dongyu LIN, Dean P. FOSTER, and Lyle H. UNGAR. 2011) From the paper, this algorithm is appears to be for larger N while your data set of only 100 N is rather small, so your mileage may vary.

require(VIF)
require(cvTools);
#returns selected variables using VIF and kfolds cross validation 
ezvif=function(df,yvar,folds=5,trace=F,ignore=c()){
  df=discard(df,ignore);
  f=cvFolds(nrow(df),K=folds);
  findings=list();
  for(v in names(df)){
    if(v==yvar)next;
    findings[[v]]=0; 
  }
  for(i in 1:folds){   
    if(trace) message("fold ",i);
    rows=f$subsets[f$which!=i] ##leave one out 
    y=df[rows,yvar];
    xdf=df[rows,names(df) != yvar]; #remove output var    
    if(trace) say("trying ",i,yvar,nrow(df),length(y)," subsize=",min(200,floor(nrow(xdf))));
    vifResult=vif(y,xdf,trace=trace,subsize=min(200,floor(nrow(xdf))))
    if(trace) print(names(xdf)[vifResult$select]);
    for(v in names(xdf)[vifResult$select]){
      findings[[v]]=findings[[v]]+1; #vote
    }
  }
  findings=(sort(unlist(findings),decreasing = T))    
  if(trace) print(findings[findings>0]); 
  return( c(yvar,names(findings[findings==findings[1]])) )  
}

#converts ezvif results into formula
ezformula=function(v,operator=' + '){
  return(as.formula(paste(v[1],'~',paste(v[-1],collapse = operator))))
}

The code I put together uses cross validation which could address your smaller n situation.

You think you nearly have a big p small n problem.

You may want to read Summary of "Large p, Small n" results and "Variable Selection for Classification and Regression in Large p, Small n Problems" (http://www.stat.wisc.edu/~loh/treeprogs/guide/lchen.pdf).

I hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.