Is there a way to use cross validation to do variable/feature selection in R? I have a data set with about 70 variables that I'd like to cut down.  What I'm looking to do is use CV to find most useful variables in the following fashion.  
1) Randomly select say 20 variables.  
2) Use stepwise/LASSO/lars/etc to choose most important variables.
3) Repeat ~50x and see which variables are selected (not eliminated) most frequently.
This is along the lines of what a randomForest would do, but the rfVarSel package seems to only work for factors/classification and I need to predict a continuous dependent variable.
I'm using R so any suggestions would ideally be implemented there.
 A: I believe what you describe is already implemented in the caret package.  Look at the rfe function or the vignette here: http://cran.r-project.org/web/packages/caret/vignettes/caretSelection.pdf
Now, having said that, why do you need to reduce the number of features?  From 70 to 20 isn't really an order of magnitude decrease.  I would think you'd need more than 70 features before you would have a firm prior believe that some of the features really and truly don't matter.  But then again, that's where a subjective prior comes in I suppose.
A: There is no reason why variable selection frequency provides any information that you do not already get from the apparent importance of the variables in the initial model.  This is essentially a replay of initial statistical significance.  you are also adding a new level of arbitrariness when trying to decide on a cutoff for selection frequency.  Resampling variable selection is badly damaged by collinearity in addition to the other problems.
A: I have revised my answer from earlier today. I have now generated some example data on which to run the code. Others have rightly suggested that you look into using the caret package, which I agree with. In some instances, however, you may find it necessary to write your own code. Below I have attempted to demonstrate how to use the sample() function in R to randomly assign observations to cross-validation folds. I also use for loops to perform variable pre-selection (using univariate linear regression with a lenient p value cutoff of 0.1) and model building (using stepwise regression) on the ten training sets. You can then write your own code to apply the resultant models to the validation folds. Hope this helps!
################################################################################
## Load the MASS library, which contains the "stepAIC" function for performing
## stepwise regression, to be used later in this script
library(MASS)
################################################################################


################################################################################
## Generate example data, with 100 observations (rows), 70 variables (columns 1
## to 70), and a continuous dependent variable (column 71)
Data <- NULL
Data <- as.data.frame(Data)

for (i in 1:71) {
for (j in 1:100) {
Data[j,i]  <- rnorm(1) }}

names(Data)[71] <- "Dependent"
################################################################################


################################################################################
## Create ten folds for cross-validation. Each observation in your data will
## randomly be assigned to one of ten folds.
Data$Fold <- sample(c(rep(1:10,10)))

## Each fold will have the same number of observations assigned to it. You can
## double check this by typing the following:
table(Data$Fold)

## Note: If you were to have 105 observations instead of 100, you could instead
## write: Data$Fold <- sample(c(rep(1:10,10),rep(1:5,1)))
################################################################################


################################################################################
## I like to use a "for loop" for cross-validation. Here, prior to beginning my
## "for loop", I will define the variables I plan to use in it. You have to do
## this first or R will give you an error code.
fit <- NULL
stepw <- NULL
training <- NULL
testing <- NULL
Preselection <- NULL
Selected <- NULL
variables <- NULL
################################################################################


################################################################################
## Now we can begin the ten-fold cross validation. First, we open the "for loop"
for (CV in 1:10) {

## Now we define your training and testing folds. I like to store these data in
## a list, so at the end of the script, if I want to, I can go back and look at
## the observations in each individual fold
training[[CV]] <- Data[which(Data$Fold != CV),]
testing[[CV]]  <- Data[which(Data$Fold == CV),]

## We can preselect variables by analyzing each variable separately using
## univariate linear regression and then ranking them by p value. First we will
## define the container object to which we plan to output these data.
Preselection[[CV]] <- as.data.frame(Preselection[CV])

## Now we will run a separate linear regression for each of our 70 variables.
## We will store the variable name and the coefficient p value in our object
## called "Preselection".
for (i in 1:70) {
Preselection[[CV]][i,1]  <- i
Preselection[[CV]][i,2]  <- summary(lm(Dependent ~ training[[CV]][,i] , data = training[[CV]]))$coefficients[2,4]
}

## Now we will remove "i" and also we will name the columns of our new object.
rm(i)
names(Preselection[[CV]]) <- c("Variable", "pValue")

## Now we will make note of those variables whose p values were less than 0.1.
Selected[[CV]] <- Preselection[[CV]][which(Preselection[[CV]]$pValue <= 0.1),] ; row.names(Selected[[CV]]) <- NULL

## Fit a model using the pre-selected variables to the training fold
## First we must save the variable names as a character string
temp <- NULL
for (k in 1:(as.numeric(length(Selected[[CV]]$Variable)))) {
temp[k] <- paste("training[[CV]]$V",Selected[[CV]]$Variable[k]," + ",sep="")}
variables[[CV]] <- paste(temp, collapse = "")
variables[[CV]] <- substr(variables[[CV]],1,(nchar(variables[[CV]])-3))

## Now we can use this string as the independent variables list in our model
y <- training[[CV]][,"Dependent"]
form <- as.formula(paste("y ~", variables[[CV]]))

## We can build a model using all of the pre-selected variables
fit[[CV]] <- lm(form, training[[CV]])

## Then we can build new models using stepwise removal of these variables using
## the MASS package
stepw[[CV]] <- stepAIC(fit[[CV]], direction="both")

## End for loop
}

## Now you have your ten training and validation sets saved as training[[CV]]
## and testing[[CV]]. You also have results from your univariate pre-selection
## analyses saved as Preselection[[CV]]. Those variables that had p values less
## than 0.1 are saved in Selected[[CV]]. Models built using these variables are
## saved in fit[[CV]]. Reduced versions of these models (by stepwise selection)
## are saved in stepw[[CV]].

## Now you might consider using the predict.lm function from the stats package
## to apply your ten models to their corresponding validation folds. You then
## could look at the performance of the ten models and average their performance
## statistics together to get an overall idea of how well your data predict the
## outcome.
################################################################################

Before performing cross-validation, it is important that you read about its proper use. These two references offer excellent discussions of cross-validation:


*

*Simon RM, Subramanian J, Li MC, Menezes S. Using cross-validation to evaluate predictive accuracy of survival risk classifiers based on high-dimensional data. Brief Bioinform. 2011 May;12(3):203-14. Epub 2011 Feb 15. http://bib.oxfordjournals.org/content/12/3/203.long

*Richard Simon, Michael D. Radmacher, Kevin Dobbin and Lisa M. McShane. Pitfalls in the Use of DNA Microarray Data for Diagnostic and Prognostic Classification. JNCI J Natl Cancer Inst (2003) 95 (1): 14-18. http://jnci.oxfordjournals.org/content/95/1/14.long
These papers are geared toward biostatisticians, but would be useful for anyone.
Also, always keep in mind that using stepwise regression is dangerous (although using cross-validation should help to alleviate overfitting). A good discussion of stepwise regression is available here: http://www.stata.com/support/faqs/stat/stepwise.html.
Let me know if you have any additional questions!
A: I just found something nice over here: http://cran.r-project.org/web/packages/Causata/vignettes/Causata-vignette.pdf
Try this maybe when using the glmnet Package
# extract nonzero coefficients
coefs.all <- as.matrix(coef(cv.glmnet.obj, s="lambda.min"))
idx <- as.vector(abs(coefs.all) > 0)
coefs.nonzero <- as.matrix(coefs.all[idx])
rownames(coefs.nonzero) <- rownames(coefs.all)[idx]

