Inferring feature importance on thousands of features for classification problem

Question

I want to assess the importance/association of many features (continuous) with a Y variable (continuous and, in most cases, skewed/normally distributed). So far I have tried running linear regression and extracted the beta coefficients for each X variable against Y. However, I am wondering if I could approach this task differently in a way that would not require linearity. Note, I have a thousand Y variables, and want to associate these with dataframe X with 1000s of columns (each Y variable independently), systematically.

An approach I've considered is turning this into a classification problem, where I split the Y variable into high (1) and low (0) classes, and run logistic regression with the complete X dataset, and use the logistic regression coefficients to determine the contribution of each feature to the identification of Y. The aim would be to trim down the dataset with "important" features which can identify Y. Alternatively, I was thinking about performing this with random forest, and identifying feature importance through this approach.

Would an approach like this make sense, or is this overkill? As a more direct method I'm simply calculating the fold difference between classes 1 and 0, but wanted to explore other methods for this, which could potentially spot feature interactions. Note at this stage I have removed redundant X features such as those with many missing values/low variance.

EDIT

The aim of this is similar to that in a machine learning setting where the features that give the most predictive power are used to boost model performance. Note however that I am not building a model for prediction use, simply to identify features/combination of features that influence Y.

I should also add that there is a great imbalance between the number of observations (X rows) and features (~3000 X cols). It varies between datasets, but the observation number is between 300-800, but can be as low as 30. Thus, I do not expect to build a model with high predictive power, but simply identify any interesting features.

I have tried this using the Y variable split mentioned above, and random forest feature importance, which does identify some features which show difference between the two Y conditions. I'm aware of the problem of overfitting, especially where I have less data.

A question I have on this is: Considering I am only using the random forest for identifying important features, and not external predictions, is overfitting as much of an issue as it is in the conventional sense?

EDIT 2

mkt made a good point on defining variable importance. Reading a suggested thread What are variable importance rankings useful for? highlighted some interesting points for me.

In my case, I wish to identify key features in relation to Y that I can identify and study in external datasets. This is not feasible with >1000 features, especially as many will have no observable relation to Y.

Answer 1

If you had a large number of data points, I'd strongly recommend simply fitting a random forest while keep your response continuous. Random forests can deal with possible nonlinearities and are structurally quite robust to overfitting. There's no need to dichotomise your continuous variable - it throws away information that is likely useful - so just keep it continuous. Importance can be defined in different ways (more on this in a bit), but the 'permutation importance' commonly used for random forests is conceptually appealing.

But with 300-800 data points and >3000 predictors, fitting any flexible model is optimistic. You may be better off fitting a less-flexible model such as a linear regression, though I suppose you could include quadratic terms if linearity is too strong an assumption. An important part of this fitting would be to use regularisation, such as LASSO. Note however that the retained parameters aren't necessarily more important, in part because importance isn't a single well-defined concept. The relaimpo R package implements a variety of different importance metrics and they can provide fairly different rankings.

You could try both modelling options and see how well they work in your case. For importance rankings, I recommend looking into the different metrics and thinking carefully about what way of quantifying importance would be most useful for your specific application.