Identifying features using two-sample t-test - resulting in over-fitting? Let's say that I have a binary classification problem, thousands of features. If I were to run a two-sample t-test on some of these features to figure out which ones differ significantly between classes (correcting for multiple-comparisons), and then use the significant features in an SVM model, will this result in substantial over-fitting? If so, what are alternative ways I can sub-select a group of features that best predict classes while maintaining generalization ability?
 A: Selecting features by univariate comparisons is not the best way to do things. If you're not careful you can end up selecting a terrible model. Even if you don't do that, it is often the case that the best combinations of features are not the best features individually. 
Here's an example to consider to show how these $t$-tests can lead you astray:
Suppose we have features $X_j = (X_{1j}, ..., X_{nj}) \sim \ \text{iid} \ \mathcal N(\vec 0, I_n)$ and labels $Y_i \sim \ \text{iid} \ Bern({1 \over 2})$, with $X_{j} \perp Y_i$ for all $i,j$. Say $i = 1, ..., n$ and $j = 1, ..., p$. Note that everything is independent so there should be no good model here.
Let's say that we perform $p$ two-sample $t$-tests and take the 10 predictors with the most significant tests. Just by chance there will be some predictors that seem fairly related to $Y$ (to be more precise, every null hypothesis is false but if we do enough tests we'll make a lot of type I errors). But clearly the variables that we've selected this way still aren't related to the response. If we found the cross-validated error rate (CVER) for this model AFTER doing the variable selection it would look pretty good, but this model would do horribly for a new observation because there isn't actually any relationship here. We've just found features that by chance seemed to be related in this sample. You could avoid this by cross-validating the entire model selection procedure. If you did that then you would see that you haven't found any good features.
Now here's an example where the features marginally are useless but together perfectly separate the response. Let $X_1, X_2 \sim \ \text{iid} \ Bern(1/2)$ and let $Y = I(X_1 \neq X_2)$, i.e. $Y$ is the indicator that $X_1$ and $X_2$ are not equal. Now we'll find that either $X_j$ alone is not helpful but together they perfectly predict $Y$. In R:
set.seed(1)
n <- 1000
x1 <- rbinom(n, 1, .5)
x2 <- rbinom(n, 1, .5)
y <- factor(as.numeric(x1 != x2))

library(rpart) ## simple classification tree
m1 <- rpart(y ~ x1) ## doesn't do well
m2 <- rpart(y ~ x2) ## doesn't do well
m12 <- rpart(y ~  x1 + x2) ## does perfectly

How do we then properly select features?
One of the simplest ways is to use a stepwise regression. For SVMs there's recursive feature elimination which is similar in spirit. Another popular method is the lasso. You can also employ the grouped lasso to select or exclude multiple variables together (for instance, instead of selecting individual levels of a factor you could select or exclude the entire variable at once). You can probably find out more about these elsewhere on the site.
