Feature selection in multi class environment I have a $\mathbb{R}^{10000 \times 25000}$ feature matrix (10000 observations and 25000 features). The observations come from 4 different classes, i.e. it is a multiclass classification problem. I want to rank the features and reduce them accordingly to their importance. A feature reduction method like PCA is not relevant for me because I have to maintain the information about the features.
I found a few methods like, $\chi^{2}$, ANOVA, Pearson correlation, and so on...
But I haven't really figured out yet which is the best method for this application. My choice would probably have been ANOVA (scikit-learn selectKBest with f_classif)
However, I have noted from different sources that this method is mainly used when the feature is categorical and the target variable is continuous. For me it is the other way around. Also I couldn't find any information if this method works with multi-class classification.
Hence the question, has anyone already experienced such a constellation and can recommend a method? Or can ANOVA be used in this case?
 A: *

*None of the methods you mention are used for feature selection.

*You have high-dimensional data (where $p>n$), so regular methods go out the window.

*Given your data, and the fact that you want to keep your variables intact and reduce the dimensionality, then something like LASSO is more appropriate, or a Random Forest.

A: It is very unlikely to run feature selection on such many features.
Before, you do feature selection, you may ask why? In many modern machine learning models people are not doing feature selection at all, but use all of them to have better accuracy.
You may want to do feature selection for reasons like increasing interpretability. But if you say you have 25K features, I am guessing the data  may be images or one-hot encoded texts. For such data, feature selection you mentioned are not widely used.
As @user2974951 mentioned, if you have n features larger than n data points, regularization should be used.
A: Create a smaller matrix, with syntetic dependence between class and features. (i.e. class=quantile (feature1+2*feature2+3*feature3+..) 
Generate syntetic features which have similar distibution to your data.
Run all Univariate selection methods you want to check which one works the best on syntetic data, and use it on real data.
