Subset selection features acquired from randomized logistic regression

Question

I learned about the concept of randomized logistic regression(or randomized lasso) recently.

My data, biological data called Microarray, usually has large features but small samples - 10000 features but 500 samples. Here features means genes, and sample means tissues from patients, which is divided into Tumor/Normal.

I want to find important genes related to tumor class. It is reported that general feature selection such as univariate, it is possible to find genes important for classification performance, but they cannot guaratee robustness. (Stability selection, Meinshausen et al.) If samples are slightly changed, the selected genes changes completely.

So I am using randomized logistic regression, which make several bootstrap samples containing slightly changed data. Then it selects features from each bootstrap and aggregates them into a full feature list. It uses lasso penatly, so features which thought to be unnecessary are deleted automatically.

This methods reduce the number of genes dramatically - similar to the number of samples(10000->500). But still, there are too many genes for tumor analysis. I want only tens of genes, so I want to find good subset from it. Methods that I am thinking of these 4 ideas.

1. Set cut-off moderately
2. Use conventional feature selection for it
3. Use RFECV (recursive feature elimination cross validation) of python. It evaluates 10-fold cross-validated prediction score such as ROC while eliminating feauture one by one using RFE.
4. Use aggregated score of randomized logistic regression. Similar to 2, but calculate 10-fold cross-valdation prediction score while eliminating feature one by one, using aggregated score acquired previous step. (CV score->Delete the gene of least score->CV score->...)

Which one will be better to reduce features, maintaining both robustness and performance? Or is there more better way to achieve it? Thank you!

Answer 1

In similar situations, I use directly LASSO with the whole sample, providing a regression model with few genes and good predictive capability. Using the glmnet() function of R, you could select the lambda parameter for penalization with the minimun performance or applying the 1-SE rule, giving a model with less variables.

The instability of the models, doing small changes in the sample, is an habitual situation with high-dimensionality data. When we have a lot of variables, there are several "best models", with similar predictive capability. This is relevant with genetic data, where the genes are correlated.

Answer 2

One simple method for feature selection would be calculating the correlation of each variable with target. Thus you get a list of variables sorted by correlation.

• If you don't expect your variables to be correlated with each other, the list provides you variable importance in a sense. You can directly take out the top variables from the list.

• In case your variables are correlated, you can take a generous number of variables and then shove than into a model which accounts for correlation between variables. From here you can identify the most important variables.

You might have to do some bootstrapping for the second case you take >500 variables because there a lot of correlation between variables themselves.