I want to assess the importance/association of many features (continuous) with a Y variable (continuous and, in most cases, skewed/normally distributed). So far I have tried running linear regression and extracted the beta coefficients for each X variable against Y. However, I am wondering if I could approach this task differently in a way that would not require linearity. Note, I have a thousand Y variables, and want to associate these with dataframe X with 1000s of columns (each Y variable independently), systematically.
An approach I've considered is turning this into a classification problem, where I split the Y variable into high (1) and low (0) classes, and run logistic regression with the complete X dataset, and use the logistic regression coefficients to determine the contribution of each feature to the identification of Y. The aim would be to trim down the dataset with "important" features which can identify Y. Alternatively, I was thinking about performing this with random forest, and identifying feature importance through this approach.
Would an approach like this make sense, or is this overkill? As a more direct method I'm simply calculating the fold difference between classes 1 and 0, but wanted to explore other methods for this, which could potentially spot feature interactions. Note at this stage I have removed redundant X features such as those with many missing values/low variance.
The aim of this is similar to that in a machine learning setting where the features that give the most predictive power are used to boost model performance. Note however that I am not building a model for prediction use, simply to identify features/combination of features that influence Y.
I should also add that there is a great imbalance between the number of observations (X rows) and features (~3000 X cols). It varies between datasets, but the observation number is between 300-800, but can be as low as 30. Thus, I do not expect to build a model with high predictive power, but simply identify any interesting features.
I have tried this using the Y variable split mentioned above, and random forest feature importance, which does identify some features which show difference between the two Y conditions. I'm aware of the problem of overfitting, especially where I have less data.
A question I have on this is: Considering I am only using the random forest for identifying important features, and not external predictions, is overfitting as much of an issue as it is in the conventional sense?
mkt made a good point on defining variable importance. Reading a suggested thread What are variable importance rankings useful for? highlighted some interesting points for me.
In my case, I wish to identify key features in relation to Y that I can identify and study in external datasets. This is not feasible with >1000 features, especially as many will have no observable relation to Y.