Useful methods to find out variable importance? I have data with 53 records and 52 variables and want to find a suitable predictive model. I think it makes sense to do some dimension reduction and select only a subset of predictors. My data contain 7 categorical predictors. The rest of the variables are numeric but not independent and with different scaling/distribution. 
My question:
Which method is useful for this structure of data to find out variable importance and reduce dimension (PCA, Random Forest, MARS)?
 A: Finding variable importance and reduce dimension are different tasks. You can :


*

*Rank the variable according to the importance you suspect they have with respect to a target (sometimes referred as filtering). This could be: only retain the predictors correlated to the target, or rank them from a random forest importance scoring and keep the highest $n$% most important variables.

*Perform a "blind" dimension reduction, regardless of the target (random projections, PCA...)


As you don't have many examples, I suspect that filtering will lead to overfit. As your numeric predictors are correlated, scaling and dimension reduction with PCA will be a first step. 
As for the categorical predictors, just remove the scarce levels if there are any. Per example, say you decomposed a variable age in levels (20-29,30-39,40-49,50-59,60+) and you discover that the category 60 years+ is represented only once. Then it will be better to merge it with the category 50-59 (calling it 50+) so that this category has more observations (and you dropped a category with only one observations).
