1
$\begingroup$

I have a $\mathbb{R}^{10000 \times 25000}$ feature matrix (10000 observations and 25000 features). The observations come from 4 different classes, i.e. it is a multiclass classification problem. I want to rank the features and reduce them accordingly to their importance. A feature reduction method like PCA is not relevant for me because I have to maintain the information about the features.

I found a few methods like, $\chi^{2}$, ANOVA, Pearson correlation, and so on...

But I haven't really figured out yet which is the best method for this application. My choice would probably have been ANOVA (scikit-learn selectKBest with f_classif)

However, I have noted from different sources that this method is mainly used when the feature is categorical and the target variable is continuous. For me it is the other way around. Also I couldn't find any information if this method works with multi-class classification.

Hence the question, has anyone already experienced such a constellation and can recommend a method? Or can ANOVA be used in this case?

$\endgroup$

3 Answers 3

1
$\begingroup$
  1. None of the methods you mention are used for feature selection.
  2. You have high-dimensional data (where $p>n$), so regular methods go out the window.
  3. Given your data, and the fact that you want to keep your variables intact and reduce the dimensionality, then something like LASSO is more appropriate, or a Random Forest.
$\endgroup$
2
  • $\begingroup$ "None of the methods you mention are used for feature selection" I think this statement is incorrect... $\endgroup$
    – Haitao Du
    Jul 7, 2021 at 10:25
  • $\begingroup$ @HaitaoDu Well, I am sure that some of them are used for this purpose, but that is not their purpose, was my point. PCA is for feature reduction, a chi-square test is for global tests, while an ANOVA or a correlation test returns p-values by variables, which one might use to "select" variables, but that is not the purpose of p-values. $\endgroup$ Jul 7, 2021 at 10:30
1
$\begingroup$

It is very unlikely to run feature selection on such many features.

Before, you do feature selection, you may ask why? In many modern machine learning models people are not doing feature selection at all, but use all of them to have better accuracy.

You may want to do feature selection for reasons like increasing interpretability. But if you say you have 25K features, I am guessing the data may be images or one-hot encoded texts. For such data, feature selection you mentioned are not widely used.

As @user2974951 mentioned, if you have n features larger than n data points, regularization should be used.

$\endgroup$
0
$\begingroup$

Create a smaller matrix, with syntetic dependence between class and features. (i.e. class=quantile (feature1+2*feature2+3*feature3+..)

Generate syntetic features which have similar distibution to your data.

Run all Univariate selection methods you want to check which one works the best on syntetic data, and use it on real data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.