I have a matrix $x=N\times M$ of $N$ data points, where each one has $M$ features. Also, $y$ is the binary labels vector. In my case, $N$ is much smaller than $M$, so before running a classifier like SVM I used PCA to reduce the dimensionality. I picked the first 100 components which explained 85% of the variance. SVM with those 100 features gave me a poor discrimination of AUC ~0.6. Recently I thought on another method for dimensionality reduction, which I'm not sure about its validity: Each feature is a vector of $N$ values, which can be separated into 2 different vectors according to the labels. So for each feature, I can calculate how well it can differentiate the two classes by its own, using AUC, or even just the p-value of a t-test (given the histograms are normal). The problem with this method, is the dependency of the features vectors. Does anyone know how can I pick the best $k$ features vectors, which together differentiate the classes in the best way?