Can feature selection be considered a way to observe relationship between variables like correlation?

In correlation we can observe relationship between a pair of variables, let me call it X1 and Y.

Now, considering I have the predicting variables X1, X2, ..., Xn and the variable Y. Does the following assumption holds:

If the variables Xi,...,Xj are observed to better classify Y, where 1 <= i,j <= n, can I claim that the variables Xi,...,Xj have more association with Y rather the remaining variables Xk, k not belonging to the i,j interval.

• It may well depend on how feature selection is done. But if you think of univariate screening based on correlation tests (with, e.g., Bonferroni or FDR correction) you will not recover variables that are not correlated to Y but have a non-zero partial correlation.
– chl
Nov 14, 2012 at 22:18
• Matlab employs a method that I found simple and intuitive for feature selection which is to permute the values of a given variable while keeping the others constant and do that for each variable. The SSE is then observed if it increases or decreases and I believe divided by the standard deviation error. This gives a measure of importance of the variable. Do you think that for this metric this could be seen as an alternative for correlation? I will see on the methods you mentioned. Thanks. Nov 14, 2012 at 22:23

In a word yes. In a Bayesian framework, a good feature (or feature set) $X$ is one where $P(Y | X )$ depends significantly on the specific value of $X$ observed; which can be reasonably interpreted as the variables being correlated. One way to quantify this is in terms of the relative entropy between the distributions.