3
$\begingroup$

I want to assess the importance/association of many features (continuous) with a Y variable (continuous and, in most cases, skewed/normally distributed). So far I have tried running linear regression and extracted the beta coefficients for each X variable against Y. However, I am wondering if I could approach this task differently in a way that would not require linearity. Note, I have a thousand Y variables, and want to associate these with dataframe X with 1000s of columns (each Y variable independently), systematically.

An approach I've considered is turning this into a classification problem, where I split the Y variable into high (1) and low (0) classes, and run logistic regression with the complete X dataset, and use the logistic regression coefficients to determine the contribution of each feature to the identification of Y. The aim would be to trim down the dataset with "important" features which can identify Y. Alternatively, I was thinking about performing this with random forest, and identifying feature importance through this approach.

Would an approach like this make sense, or is this overkill? As a more direct method I'm simply calculating the fold difference between classes 1 and 0, but wanted to explore other methods for this, which could potentially spot feature interactions. Note at this stage I have removed redundant X features such as those with many missing values/low variance.

EDIT

The aim of this is similar to that in a machine learning setting where the features that give the most predictive power are used to boost model performance. Note however that I am not building a model for prediction use, simply to identify features/combination of features that influence Y.

I should also add that there is a great imbalance between the number of observations (X rows) and features (~3000 X cols). It varies between datasets, but the observation number is between 300-800, but can be as low as 30. Thus, I do not expect to build a model with high predictive power, but simply identify any interesting features.

I have tried this using the Y variable split mentioned above, and random forest feature importance, which does identify some features which show difference between the two Y conditions. I'm aware of the problem of overfitting, especially where I have less data.

A question I have on this is: Considering I am only using the random forest for identifying important features, and not external predictions, is overfitting as much of an issue as it is in the conventional sense?

EDIT 2

mkt made a good point on defining variable importance. Reading a suggested thread What are variable importance rankings useful for? highlighted some interesting points for me.

In my case, I wish to identify key features in relation to Y that I can identify and study in external datasets. This is not feasible with >1000 features, especially as many will have no observable relation to Y.

$\endgroup$
6
  • $\begingroup$ Welcome to Cross Validated! Why do you have to trim down the feature set at all? $\endgroup$
    – Dave
    May 30, 2023 at 23:55
  • $\begingroup$ Hi Dave, I am trying to determine which features have the strongest relationship with Y, akin to feature selection in a machine learning setting, though at this stage I am not trying to develop a model for prediction use. Out the approx 3000 features most of them have no relationship with Y, and so are not useful to me. I have edited my post to include some more information and progress. Many thanks $\endgroup$
    – LJM
    Jun 1, 2023 at 7:37
  • 1
    $\begingroup$ You could use the random forest and permutation importance while keep your response continuous. There's no need to dichotomise your continuous variable, and good reasons to avoid doing that. $\endgroup$
    – mkt
    Jun 1, 2023 at 7:54
  • $\begingroup$ Note that there are multiple ways to define 'importance'. It's worth looking at some threads we have on the topic: stats.stackexchange.com/questions/tagged/importance?tab=Votes $\endgroup$
    – mkt
    Jun 1, 2023 at 7:56
  • $\begingroup$ @mkt thanks for this. A good point on dichotomisation, whilst it is intuitive in the data I am using, there will be differences in the variance (quite large in some cases) of each Y variable, so keeping the variable continuous could retain key information in the data, if I am not mistaken. $\endgroup$
    – LJM
    Jun 1, 2023 at 9:14

1 Answer 1

2
$\begingroup$

If you had a large number of data points, I'd strongly recommend simply fitting a random forest while keep your response continuous. Random forests can deal with possible nonlinearities and are structurally quite robust to overfitting. There's no need to dichotomise your continuous variable - it throws away information that is likely useful - so just keep it continuous. Importance can be defined in different ways (more on this in a bit), but the 'permutation importance' commonly used for random forests is conceptually appealing.

But with 300-800 data points and >3000 predictors, fitting any flexible model is optimistic. You may be better off fitting a less-flexible model such as a linear regression, though I suppose you could include quadratic terms if linearity is too strong an assumption. An important part of this fitting would be to use regularisation, such as LASSO. Note however that the retained parameters aren't necessarily more important, in part because importance isn't a single well-defined concept. The relaimpo R package implements a variety of different importance metrics and they can provide fairly different rankings.

You could try both modelling options and see how well they work in your case. For importance rankings, I recommend looking into the different metrics and thinking carefully about what way of quantifying importance would be most useful for your specific application.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.