I learned about the concept of randomized logistic regression(or randomized lasso) recently.
My data, biological data called Microarray, usually has large features but small samples - 10000 features but 500 samples. Here features means genes, and sample means tissues from patients, which is divided into Tumor/Normal.
I want to find important genes related to tumor class. It is reported that general feature selection such as univariate, it is possible to find genes important for classification performance, but they cannot guaratee robustness. (Stability selection, Meinshausen et al.) If samples are slightly changed, the selected genes changes completely.
So I am using randomized logistic regression, which make several bootstrap samples containing slightly changed data. Then it selects features from each bootstrap and aggregates them into a full feature list. It uses lasso penatly, so features which thought to be unnecessary are deleted automatically.
This methods reduce the number of genes dramatically - similar to the number of samples(10000->500). But still, there are too many genes for tumor analysis. I want only tens of genes, so I want to find good subset from it. Methods that I am thinking of these 4 ideas.
- Set cut-off moderately
- Use conventional feature selection for it
- Use RFECV (recursive feature elimination cross validation) of python. It evaluates 10-fold cross-validated prediction score such as ROC while eliminating feauture one by one using RFE.
- Use aggregated score of randomized logistic regression. Similar to 2, but calculate 10-fold cross-valdation prediction score while eliminating feature one by one, using aggregated score acquired previous step. (CV score->Delete the gene of least score->CV score->...)
Which one will be better to reduce features, maintaining both robustness and performance? Or is there more better way to achieve it? Thank you!