I have data with 53 records and 52 variables and want to find a suitable predictive model. I think it makes sense to do some dimension reduction and select only a subset of predictors. My data contain 7 categorical predictors. The rest of the variables are numeric but not independent and with different scaling/distribution. My question:

Which method is useful for this structure of data to find out variable importance and reduce dimension (PCA, Random Forest, MARS)?


Finding variable importance and reduce dimension are different tasks. You can :

  • Rank the variable according to the importance you suspect they have with respect to a target (sometimes referred as filtering). This could be: only retain the predictors correlated to the target, or rank them from a random forest importance scoring and keep the highest $n$% most important variables.
  • Perform a "blind" dimension reduction, regardless of the target (random projections, PCA...)

As you don't have many examples, I suspect that filtering will lead to overfit. As your numeric predictors are correlated, scaling and dimension reduction with PCA will be a first step.

As for the categorical predictors, just remove the scarce levels if there are any. Per example, say you decomposed a variable age in levels (20-29,30-39,40-49,50-59,60+) and you discover that the category 60 years+ is represented only once. Then it will be better to merge it with the category 50-59 (calling it 50+) so that this category has more observations (and you dropped a category with only one observations).

  • $\begingroup$ Thank you for the answer. Can you explain a bit more in detail what you mean with 'remove the scarce levels'? $\endgroup$ – R_FF92 Oct 8 '15 at 10:35
  • $\begingroup$ @fabian92 I added more details $\endgroup$ – RUser4512 Oct 8 '15 at 10:38
  • $\begingroup$ Ah ok I understand your words. But for me there is one important question left: The values of some numeric predictors are discrete or even 0/1. I treated them not as categorical because 1 should have more impact on the target as 0 (problem: in observations 90% 1 and 10% 0). Other variables are continous, for example with range 0-100. I am not sure how I should deal with those different distributions while performing a PCA $\endgroup$ – R_FF92 Oct 8 '15 at 11:05
  • $\begingroup$ What follows is just an opinion, I would just cross validate the model if I were you. But you could treat them as factors, and remove the columns is 95% (or more) are 0s (resp. 1s) $\endgroup$ – RUser4512 Oct 8 '15 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.