# Feature selection in multi class environment

I have a $$\mathbb{R}^{10000 \times 25000}$$ feature matrix (10000 observations and 25000 features). The observations come from 4 different classes, i.e. it is a multiclass classification problem. I want to rank the features and reduce them accordingly to their importance. A feature reduction method like PCA is not relevant for me because I have to maintain the information about the features.

I found a few methods like, $$\chi^{2}$$, ANOVA, Pearson correlation, and so on...

But I haven't really figured out yet which is the best method for this application. My choice would probably have been ANOVA (scikit-learn selectKBest with f_classif)

However, I have noted from different sources that this method is mainly used when the feature is categorical and the target variable is continuous. For me it is the other way around. Also I couldn't find any information if this method works with multi-class classification.

Hence the question, has anyone already experienced such a constellation and can recommend a method? Or can ANOVA be used in this case?

1. None of the methods you mention are used for feature selection.
2. You have high-dimensional data (where $$p>n$$), so regular methods go out the window.
3. Given your data, and the fact that you want to keep your variables intact and reduce the dimensionality, then something like LASSO is more appropriate, or a Random Forest.

Create a smaller matrix, with syntetic dependence between class and features. (i.e. class=quantile (feature1+2*feature2+3*feature3+..)

Generate syntetic features which have similar distibution to your data.

Run all Univariate selection methods you want to check which one works the best on syntetic data, and use it on real data.