I have a dataset which consists of around 46k observations and 20k features. The target vector is of length 75 (and so the target matrix is 46k x 75). Among the features few are categorical and others are numerical. What are the possible best approaches for feature selection for these kind of datasets?
I am aware that unsupervised techniques such as PCA could be applied here, but what other approaches could be applied. Many variables with 0 variance were initially removed.
Well, there are actually many feature selection methods in literature so you could try some of them. One interesting FS method is the model based one.
The idea is that you train a model with all the features and then calculate the feature importance (it is provided by many models e.g. in scikit-learn). You keep only those features that are in the top-N positions or those whose importance is above X. You have to select N or X manually.
Take care to do this inside a cross-validation loop. A good idea is to keep features who remain in the top-N positions for all CV folds. You can adapt this procedure to work for the multilabel classification setting.
