Intro:
I have a dataset with a classical "large p, small n problem". The number available samples n=150 while the number of possible predictors p=400. The outcome is a continuous variable.
I want to find the most "important" descriptors, i.e., those that are best candidates for explaining the outcome and helping to build a theory.
After research on this topic I found LASSO and Elastic Net are commonly used for the case of large p, small n. Some of my predictors are highly correlated and I want to preserve their groupings in the importance assessment, therefore, I opted for Elastic Net. I suppose that I can use absolute values of regression coefficients as a measure of importance (please correct me if I am wrong; my dataset is standardized).
Problem:
As my number of samples is small, how can I achieve a stable model?
My current approach is to find best tuning parameters (lambda and alpha) in a grid search on 90% of the dataset with 10-fold cross-validation averaging MSE score. Then I train the model with the best tuning parameters on the whole 90% of dataset. I am able to evaluate my model using R squared on the holdout 10% of the dataset (which account to only 15 samples).
Running repeatedly this procedure, I found a large variance in R squared assessments. As well, the number of non-zeroed predictors varies as well as their coefficients.
How can I get a more stable assessment of predictors' importance and more stable assessment of final model performance?
Can I repeatedly run my procedure to create a number of models, and then average regression coefficients? Or should I use the number of occurrences of a predictor in the models as its importance score?
Currently, I get around 40-50 non-zeroed predictors. Should I penalize number of predictors harder for better stability?
Any help will be greatly appreciated!
