package {glmnet} too many variables with Lasso

Question

I used the glmnet-package to do a regression + variable-selection with Lasso. I had n=100 oberservations and p=200 covariables. I always read that after variable-selection with the Lasso there a maximal min(n,p) covariables in the resulting model. But I got a model with 102 covariables. Is there anyone with a explanation for that result? In my opinion there should be maximal 100 variables in the model. Thanks!

Answer 1

Lasso is a linear regression with a small additional constraint: together with prediction errors, large absolute values of regression coefficients are penalized (so called l1 penalty as opposed to l2 in ridge regression). The amount of penalization is governed by regularization parameter lambda. In general, the higher the lambda the fewer coefficients are left in the model. Optimal number of coefficients, or optimal model if you wish, is checked via cross-validation.

Where did you get "maximal min(n,p)"? It does not sound right according to what I learnt in my statistical courses and my experience with lasso, as number of coefficients can be any depending on value of lambda and your data.

As a general side note, lasso models are notorious for instability, so you may wish try elasticnet, which is a mix of lasso and ridge. As usual, use cross-validation to tune parameters.

Hope this helps.

Answer 2

In cases like this, I always try another variable selection procedure and see how much they are in agreement.

VIF (https://en.wikipedia.org/wiki/Variance_inflation_factor)

Here is a paper on VIF "VIF Regression: A Fast Regression Algorithm for Large Data" (https://statistics.wharton.upenn.edu/files/?whdmsaction=public:main.file&fileID=3486) (Dongyu LIN, Dean P. FOSTER, and Lyle H. UNGAR. 2011) From the paper, this algorithm is appears to be for larger N while your data set of only 100 N is rather small, so your mileage may vary.

require(VIF)
require(cvTools);
#returns selected variables using VIF and kfolds cross validation 
ezvif=function(df,yvar,folds=5,trace=F,ignore=c()){
  df=discard(df,ignore);
  f=cvFolds(nrow(df),K=folds);
  findings=list();
  for(v in names(df)){
    if(v==yvar)next;
    findings[[v]]=0; 
  }
  for(i in 1:folds){   
    if(trace) message("fold ",i);
    rows=f$subsets[f$which!=i] ##leave one out 
    y=df[rows,yvar];
    xdf=df[rows,names(df) != yvar]; #remove output var    
    if(trace) say("trying ",i,yvar,nrow(df),length(y)," subsize=",min(200,floor(nrow(xdf))));
    vifResult=vif(y,xdf,trace=trace,subsize=min(200,floor(nrow(xdf))))
    if(trace) print(names(xdf)[vifResult$select]);
    for(v in names(xdf)[vifResult$select]){
      findings[[v]]=findings[[v]]+1; #vote
    }
  }
  findings=(sort(unlist(findings),decreasing = T))    
  if(trace) print(findings[findings>0]); 
  return( c(yvar,names(findings[findings==findings[1]])) )  
}

#converts ezvif results into formula
ezformula=function(v,operator=' + '){
  return(as.formula(paste(v[1],'~',paste(v[-1],collapse = operator))))
}

The code I put together uses cross validation which could address your smaller n situation.

You think you nearly have a big p small n problem.

You may want to read Summary of "Large p, Small n" results and "Variable Selection for Classification and Regression in Large p, Small n Problems" (http://www.stat.wisc.edu/~loh/treeprogs/guide/lchen.pdf).

I hope this helps.