I'm new to predictive models and I have a problem at hand that I need some advice with. Basically for a clinical application we want to predict the outcome of a rating scale with a model built on top of outcomes of our new measurement device. My dependent variable, a clinical rating scale, is an integer between 0 and 10 (inclusive). Unfortunately I don't have a large sample ($n \approx 100$) and I have a lot features to select from ($p \approx 120$). Also many of these features are correlated. Nearly all of the features are continuous variables. I have a separate sample for validation ($ n \approx 40$). There are several issues I'd like have your advice about:

  1. Should I go for regression or tree based methods?
  2. Should I try ensemble learning methods or I'd better stick with a single model? Which methods should I try and why?
  3. If it's better to go for a single model, how should I handle the model selection problem? Should I e.g. limit the number of predictors and go for methods like LEAPS with AIC or should I go for methods like LASSO?
  4. If ensemble methods are suggested, which methods can handle cases with small $n$ and large $p$ better?
  5. Discussing selected/influential features is important for me. Depending on the answers to previous questions, how should I go about it?

I have some understanding of regression modeling and model selection problems. I have used the bestglm package in the past. Currently I'm looking at the Caret package as it brings a large number of methods under the same interface. References get technical about the details of the models but so far I didn't find a good one to go over practical issues for problems with small n and big p. I appreciate your suggestions and help.

Thanks, AlefSin


If you use a regression model you may start with ordinal logistic regression since your dependent variable has an ordinal scale of 11 levels. Then you may want to look at the threshold values as you may find that they are equidistant (after some transformation), in which case you may go for linear regression.

Tree based methods are able to capture some non-linearities, interactions and they are very good at finding thresholds in the explanatory variables. You may be able to explain some of these by adding transformed versions of the explanatory variables to the feature set of the regression analysis. Playing with ACE or AVAS may help finding suitable transformations.

As it is important for you to discuss the influential features I recommend you to do extensive exploration with trees, regression models and graphs to understand the biology behind the data and models. I would start with your last question, understand the biology first, and then formulate a conformable model.

| cite | improve this answer | |
  • $\begingroup$ Thanks GaBorgulya. You have some great points. I read the documents of ACE and AVAS and they are quite interesting and useful. However, I don't see how some of your recommendations work when we have a model selectoin problem to solve before even knowing what regression problem we have at hand. Using the biology of the problem does not eliminate the model selection in this case since we have already done that and that's one reason many of the predictors are correlated. $\endgroup$ – AlefSin Mar 29 '11 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.