Finding best model for small volume data I know, this topic was mentioned many time at this forum.
However I failed to find any detailed suggestions.
I am currently involved in the analysis of medical data.
The way I usually did this in situation of big data volume was:
I randomly split data into training and test sets. Looked for best classification (logistic regression in my problem) model, stored parameters in the best model together with their coefficients, applied this to test set and reported the results.
My current problem is: I have dataset with small amount of cases and controls (classification classes), around 45 each. However I have about 100 predictors.
I realise that if I want to make any testing, I'll have to face Bonferroni correction (or it's analogue) which will kill any result.
However I really want to find some model with its parameters and coefficients and report the results. At the same time I want to avoid both underfitting and overfitting.
If I, for example, split data randomly as 50:50, every model, obviously, performs in the different way.
My idea was to look at all possible combinations for up to 4, for example, predictors and then for each model split data, let's say 100 times, and store average performance on test set. Thus I think I could avoid overfitting in terms of amount of parameters. However in this case I face the following problem: how could I choose coefficients for these parameters (different for different splits) and which classification result could I report?
I'd be very grateful for your suggestions!
Thanks!
 A: Look into regularized regression such as elastic net implemented in the glmnet package for R. Regularized regression of this kind is capable of handling cases where the number of coefficients to estimate is much greater than the sample size. Use glmnet to select variables to retain. If you do any cross-validation on the elastic net regularization hyperprameters, I would use the bootstrap and only tune the parameter that guides how much regularization to do while leaving the parameter that determines the weight of the L1 vs L2 norm alone. I bet with 45 observations you'll end up with one or two non-zero coefficients. Then take your selected variables and fit a regularized Bayesian model with weakly informative prior over the coefficients. The bayesglm package for R is good for this. Fit as rich a model as possible, with interaction effects and everything. The bayesglm default prior will severely penalize the interactions and likely shrink them to zero, but at least you didn't assume the interaction effects are zero. Also actually look at your data with cross tabulation and plots once you have selected the candidate variables with glmnet. And do some posterior predictive checks on whether your data violates the assumptions made by the likelihood family you used in the regression. Finally, I would not recommend building some kind of predictive model based on 45 observations. This is statistical inference you are doing here, and it is inference to guide the collection of more data and the formulation of prior distributions to use in future analyses.
A: I'd suggest leave-one-out cross-validation. With 45 observations, use 44 to fit the model and calculate the error on the 45th. By leaving out each observation in turn and calculating the average error (e.g. RMS), you should get an unbiased estimate of performance. You can repeat this for different combinations of predictors, different model classes etc. Some more background here: http://robjhyndman.com/hyndsight/crossvalidation/
One advantage of small data sets is that you can use methods that are often too computationally expensive to be useful. 
Finally, be aware that building 1000's of models and picking one with the lowest error can lead to overfitting, even if every model's error was calculated using cross-validation. The selection of one model among many can introduce a bias.
