I know, this topic was mentioned many time at this forum. However I failed to find any detailed suggestions.
I am currently involved in the analysis of medical data. The way I usually did this in situation of big data volume was: I randomly split data into training and test sets. Looked for best classification (logistic regression in my problem) model, stored parameters in the best model together with their coefficients, applied this to test set and reported the results.
My current problem is: I have dataset with small amount of cases and controls (classification classes), around 45 each. However I have about 100 predictors. I realise that if I want to make any testing, I'll have to face Bonferroni correction (or it's analogue) which will kill any result. However I really want to find some model with its parameters and coefficients and report the results. At the same time I want to avoid both underfitting and overfitting.
If I, for example, split data randomly as 50:50, every model, obviously, performs in the different way. My idea was to look at all possible combinations for up to 4, for example, predictors and then for each model split data, let's say 100 times, and store average performance on test set. Thus I think I could avoid overfitting in terms of amount of parameters. However in this case I face the following problem: how could I choose coefficients for these parameters (different for different splits) and which classification result could I report?
I'd be very grateful for your suggestions! Thanks!