Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

In correlation we can observe relationship between a pair of variables, let me call it X1 and Y.

Now, considering I have the predicting variables X1, X2, ..., Xn and the variable Y. Does the following assumption holds:

If the variables Xi,...,Xj are observed to better classify Y, where 1 <= i,j <= n, can I claim that the variables Xi,...,Xj have more association with Y rather the remaining variables Xk, k not belonging to the i,j interval.

share|improve this question
1  
It may well depend on how feature selection is done. But if you think of univariate screening based on correlation tests (with, e.g., Bonferroni or FDR correction) you will not recover variables that are not correlated to Y but have a non-zero partial correlation. – chl♦ Nov 14 '12 at 22:18
Matlab employs a method that I found simple and intuitive for feature selection which is to permute the values of a given variable while keeping the others constant and do that for each variable. The SSE is then observed if it increases or decreases and I believe divided by the standard deviation error. This gives a measure of importance of the variable. Do you think that for this metric this could be seen as an alternative for correlation? I will see on the methods you mentioned. Thanks. – Oeufcoque Penteano Nov 14 '12 at 22:23

2 Answers

up vote 1 down vote accepted

In a word yes. In a Bayesian framework, a good feature (or feature set) $X$ is one where $P(Y | X )$ depends significantly on the specific value of $X$ observed; which can be reasonably interpreted as the variables being correlated. One way to quantify this is in terms of the relative entropy between the distributions.

share|improve this answer
So do you think that it will depend on the method I am using to evaluate this? Do you think that my answer comment to chi would be a reasonable way to interpret correlation among the variables? Thank you! – Oeufcoque Penteano Nov 14 '12 at 22:24
In a general sense yes. Depending on exactly what you do, there could be types of relationships that could be exploited for regression/classification but won't be easily determined by the data analysis approach. – Dave Nov 14 '12 at 22:32
up vote 0 down vote

A variable that achieves a better classification performance is not necessarily a variable that is correlated or relevant to the target class labels (i.e. Y in your case). So, optimality does not imply relevance and vice versa.

If the relevance metric you are using does not apply any transformation or pre-processing to the data and independent of any learning model, then you can infer some direct relation between the selected feature and the target variable. A good metric is one that leads to selection of features related to the target variable and that are not redundant. mRMR is an example of such a metric.

On the other hand, if your evaluation metric is not independent of the classification model which is known as the wrapper model, then selection of relevant features is based on the type of the classifier. For example, Naive Bayes Classifier and Decision Trees are methods that reflects relevance between the feature and the target variable. However, SVM with an RBF kernel might not lead to the selection of relevant features but rather, the optimal ones.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.