StackExchange newcomer here...I am looking for some advice on feature selection packages in R. Specifically, I am in search of functions that can identify the best features, out of 500+ features, within a data set that is 500 or less observations. Most of what I have tried to date (i.e. Regsubsets, Genetic Algorithm, etc.) does not provide results in less than 24 hours. I need the function(s) to be efficient enough that it can be run on a desktop and produce results in a timely manner. Lastly, I need the function to find the best combination of features from a linear regression perspective (similar to Regsubsets).

Ideally, the "winner" would be able to do all that I have mentioned above on a standard desktop/laptop in these time frames:

  • 50 or less features < 1 minute
  • 250 features < 3 minutes
  • 1000 features < 10 minutes

Fyi...I am mostly concerned with data sets that have linear relationships between features. By "best" I am referring to highest Adjusted R2 and lowest RMSE.

  • $\begingroup$ Welcome to the site. Two notes: 1) Questions about coding are off topic here, so your request to find R packages may make people close the question, you could make it more general. 2) You need to define "best". $\endgroup$ – Peter Flom - Reinstate Monica Sep 22 '18 at 23:43
  • 1
    $\begingroup$ The speed of the feature selection is also going to be a function of the number of observations you have. However, your question might be flagged as too open ended. It might be a good idea to make it more specific. $\endgroup$ – Matt L. Sep 22 '18 at 23:44
  • $\begingroup$ I realize it's open-ended as I want to keep the answers general, in hopes that I will be clued into some other options. I have searched high-and-low to find feature selection methods that are fast and accurate for both wide and deep data sets...no luck. That lead me to develop my own algorithm that is faster and as accurate or more than anything I can find. Now I am hoping to benchmark it against other algorithms. $\endgroup$ – FeatureSlimming Oct 9 '18 at 0:34

As a partial answer (see my comment as well), I would say that there is no solution that gives the absolute best (defined in whatever way you choose - $R^2$, adjusted $R^2$ or whatever). This is so because there are too many possible models. Even if we don't consider interactions, 500 variables means there are $2^{500} \approx 3*10^{150}$ possible models. If each model took 1/1000 of second to run, it would take longer than the lifetime of the universe to consider them all.

People have considered the question of variable selection a lot. There are many proposed solutions. Personally, I have my doubts about any analysis that is done via a purely automatic method - whatever it is. This would be so, even if N was very large. You should be using substantive expertise.

You note that you have only N = 500. Then one common rule of thumb would limit you to no more than 50 IVs. Putting that as a limit on regsubsets would speed things up - have you done that?

Another solution is to use LASSO. This doesn't solve all the problems posed by automatic methods, but it does work fast and it does adjust the coefficients to account for model complexity.

  • 1
    $\begingroup$ Thank you for the great reply, Peter. I have learned the hard way that there is no magic recipe and I agree "automatic" feature selection is not the answer. My goal has been to create an algorithm that will very quickly identify the "best" features and then produce numerous linear models to choose from. I have been able to produce stellar results with linear problems but I have to keep the number of feature combinations lower; otherwise, I run into too many combinations and the amount of time it takes to produce results grows exponentially - for very little gain in AdjR2 or RMSE. $\endgroup$ – FeatureSlimming Oct 9 '18 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.