Variable Selection for a binary classification problem I have a binary response variable (0,1) with 300 (approximately) normally distributed variables as predictors all on (0,1).  The training set has only 250 observations and the test set has approx. 20k observations.  The goal is to make a model to best classify the test set observations as either 0 or 1.  (it's an approximately balanced training set, with around 125 each 0 and 1).  With 300 variables and only 250 observations, I'm attempting to do a little bit of variable reduction.  First I simply checked the correlation between the predictor and each of the response variable, to see if there were any variables that were not terribly correlated with the response (relative to the others):

there doesn't seem to be any real break where I could reasonably say these variables are much more correlated with the response and i'll toss the rest.  So I tried a PCA analysis:
train_pca <- prcomp(xVar)
plot(sort(train_pca$sdev), main = "PCA$sdev graph")
sort(apply(abs(train_pca$rotation[,1:25]), 1, sum))
plot(sort(apply(abs(train_pca$rotation[,1:25]), MARGIN = 1, sum)), ylim = c(0.7,2.1), main = "PCA Rotation Graph")

I graph the rotation of each of the vectors to see again if there are any that are much greater that explain the variation than the others.

Again not any big break where I can decide that certain variables are better than the others.
Lastly I built a random forest with the importance control on:
randy_1 <- randomForest(y = yVar, x = train[,5:304], importance = TRUE, ntree = 1000)

and made an importance graph to look at the variables:

There appear to a few variables that appear better than others, but if you look at the mean gini reduction for all the variables, there really is just a uniform descent.  So basically at this point I'm more or less looking to either (1) use all the variables which isn't right (I don't think), b/c the question I've been posed suggests that variable reduction was necessary, or (2) pick a point that seems sort of arbitrary to cut off which variables to use and which to not consider (either the top half of the PCA or top half of random forest importance variables for example).  Anyone have any other ideas for variable selection that I haven't considered but should be?  Thanks.
 A: In a case such as this where the number of variables exceeds the number of observations in the training dataset, I would turn to one of the penalized regression techniques, such as lasso or elastic net, which perform both variable selection and coefficient shrinkage. Logistic regression models using lasso or elastic net can be run with the glmnet R package.
A good source on lasso and elastic net is Hastie, Tibshirani, and Friedman's text "The Elements of Statistical Learning: Data Mining, Inference, and Prediction" available online here:
http://www-stat.stanford.edu/~tibs/ElemStatLearn/
A: I'd post this as comment, but I don't have the privileges. I'm guessing you took this problem from the Kaggle Competition. Have you tried looking at what the eventual winners for the competition did? They all post their methods in that thread. Also, a google search turned up this which looked like it could be helpful at a glance.
I think there are some other good threads in the forum that may help you. There were different threads about Variable Selection and about AUC.
