Predictive modeling with feature selection using a small sample size? I am trying to build a predictive model for a binary classification problem. I have 200,000 features and 100 samples. I want to reduce the # of features and not over-fit the model, all while being constrained with a very small sample size.
This is currently what I'm doing:
from sklearn.feature_selection import RFECV
from sklearn.cross_validation import train_test_split
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
import numpy as np

# remove mean and scale to unit variance 
scaler = StandardScaler()
scaler.fit(features)
features = scaler.transform(features)

# split our data set into training, and testing
xTrain, xTest, yTrain, yTest = train_test_split(features, classes, test_size=0.30)

# create classifier to use with recursive feature elimination
svc = SVC(kernel="linear", class_weight = 'balanced')

# run recursive feature elimination with cross-validation
rfecv = RFECV(estimator=svc, step=1, cv=4,
         scoring = 'roc_auc') # pick features using roc_auc score because we have an imbalance of classes
newTrain = rfecv.fit_transform(xTrain, yTrain)

# test model
svc.fit(newTrain, yTrain)
svc.predict(xTest)

I believe that I'm getting overly-optimistic classification accuracy, likely due to model over-fitting. 
How can I test whether I am over-fitting my model? What would be the most optimal way to feature select and generate a predictive model using such a small sample size (and large # of features)?
 A: You should have a look to elastic net regression. This technique considers the high throughput setting of your data.
http://web.stanford.edu/~hastie/TALKS/enet_talk.pdf
A: The rule of thumb for unpenalized logistic regression or binary classification schemes as you have been using is to examine no more than one feature (predictor variable) per 15 cases in the least frequent category.
With only 30 cases in your least frequent class, that means you should be examining about 2 features, not 200,000. So yes, you almost certainly have been overfitting.
Elastic net as proposed in another answer allows evaluation of more predictors because it penalizes the regression coefficients to values lower in magnitude than they would have in a standard regression. It also selects a subset of predictors while eliminating the others. But starting with 200,000 predictors makes this a difficult task.
You must be very wary. At the standard p < 0.05 cutoff, you expect 10,000 false-positive relations even if none of your features is truly related to the classification.
With so many predictors you may well have "perfect separation" where a small group of your 200,000 predictors gives perfect prediction--in this data set, but probably for no other sample from the same population. And if you use elastic net (or its limiting version, the LASSO) to select a subset of features, the particular features selected will be very sample-dependent. Try repeating the variable selection process on multiple bootstrap samples of your data to see that problem in action.
Also, your reliance on AUC values to evaluate schemes has hidden assumptions of which you might not be aware. That approach essentially assumes that all types of misclassifications are equally important. That is seldom the case. You will typically be better off building a model that predicts probabilities of class membership well, as with a penalized logistic regression, then evaluate the costs of different misclassifications to choose a probability threshold for classification.
But none of these approaches may work in a reliable way when you have a class of 30 members and 200,000 features.
