I'm new to predictive models and I have a problem at hand that I need some advice with. Basically for a clinical application we want to predict the outcome of a rating scale with a model built on top of outcomes of our new measurement device. My dependent variable, a clinical rating scale, is an integer between 0 and 10 (inclusive). Unfortunately I don't have a large sample ($n \approx 100$) and I have a lot features to select from ($p \approx 120$). Also many of these features are correlated. Nearly all of the features are continuous variables. I have a separate sample for validation ($ n \approx 40$). There are several issues I'd like have your advice about:
- Should I go for regression or tree based methods?
- Should I try ensemble learning methods or I'd better stick with a single model? Which methods should I try and why?
- If it's better to go for a single model, how should I handle the model selection problem? Should I e.g. limit the number of predictors and go for methods like LEAPS with AIC or should I go for methods like LASSO?
- If ensemble methods are suggested, which methods can handle cases with small $n$ and large $p$ better?
- Discussing selected/influential features is important for me. Depending on the answers to previous questions, how should I go about it?
I have some understanding of regression modeling and model selection problems. I have used the bestglm package in the past. Currently I'm looking at the Caret package as it brings a large number of methods under the same interface. References get technical about the details of the models but so far I didn't find a good one to go over practical issues for problems with small n and big p. I appreciate your suggestions and help.