# Select 3 features from 4000 correlated features

I have been learning machine learning for only two months, so please pardon me and tell me how to imporve it if my question is phrased in a very naive way.

The problem that I'm facing:

There is a class of $100$ compounds AB. $x_n$ is the descriptor of the $n$th compound. $\Delta E_n$ is a scalar property of the $n$th compound.

Since there is no canonical representation of a compound AB, I'm going to start from $10$ primary features like the radius of atom A and B, then construct $3900$ auxiliary features by several algebraic operations to get a descriptor consisting of $4000$ features for each compound.

I want to select three features from the $4000$ ones that can be used to best descibe $\Delta E$.

Properties of the ten primary features:

1. The features are measured in different units so that they are of different magnitudes. For physical consistency, these features will be unshifted and unscaled.

2. Some of the primary features are highly correlated.

One way that I can think of is best subset regression. However, there are $C_{4000}^{3} \approx 10^{10}$ different combinations, which is completely infeasible to do on my own MacBook (Intel-i5 CPU).

The ultimate goal of doing this feature selection is to find proper fingerprints for the class of binary compounds. So, the predictive power and the stability of the algorithm is very important.

What I mean by the stability of the algorithm (although I don't know a definition):

I draw say $50$ bootstrap samples of size $100$ from the dataset (consisting of $100$ examples) and train the model on each bootstrap samples. I expect there is not too much fluctuations in the set of three features that is selected.

• One possibility is to perform one linear regression for each feature, 4000 in total, and keep the top 100 features with the best individual fits. This could run overnight, and would greatly reduce the size of the subset regression to something that might run over a few days. This could be worth a try. – James Phillips Sep 5 '18 at 15:55
• 1) Since you mentioned best subset regression, does this mean you want to fit $\Delta E$ with a linear function of the features? 2) Can you say more about what it means to "find proper fingerprints?" and how you want to use this information (e.g. to predict $\Delta E$ for new compounds, to learn something about the existing ones by inferring an underlying model, etc.)? How you approach the problem may depend on that. – user20160 Sep 6 '18 at 0:18
• 3) You mentioned predictive power and stability. If the only goal is prediction, then stability doesn't matter. Some feature selection methods may be unstable due to correlations in the features. But, because the features carry redundant information, different choices of features may predict equally well. But, if the goal is to infer a 'true' underlying model, then stability matters more. – user20160 Sep 6 '18 at 0:18
• @ user20160 I will edit the post to try to answer your really helpful questions, thank you! – meTchaikovsky Sep 6 '18 at 1:09