11
$\begingroup$

I have three features that I use to solve a classification problem. Originally, these features produced boolean values, so I could evaluate their redundancy by looking at how much the sets of positive and negative classifications overlap. Now I have extended the features to produce real values (scores) instead, and I would like to analyze their redundancy again, but I am at a complete loss on how to do that. Can anyone provide me with a pointer or idea on how to go about that?

I know this question is very vague, that is because I do not have a very strong grasp of statistics. So, if you do not have an answer for me, maybe you have some questions that can help me understand better myself.

Edit: I am currently browsing Wikipedia on the subject, I have the feeling that what I want is a correlation coefficient, but I am still unsure if this is the right approach, and which of the many available coefficients is appropriate.

Edit 2: In the boolean case, I first created for each feature the set of samples for which it was true. Then, the correlation between two features was the size of intersection of these sets over the size of the union of these sets. If this value is 1, they are completely redundant, because always the same. If it is 0, they are never the same.

Improve this question
$\endgroup$
2
  • $\begingroup$ it would help if you provided example of how do you define redundancy in the boolean case, and what kind of results you would expect in continuous case $\endgroup$
    – mpiktas
    Commented Feb 10, 2011 at 17:16
  • $\begingroup$ @mpiktas: Edit my question in response to your comment. $\endgroup$
    – Björn Pollex
    Commented Feb 10, 2011 at 17:35

1 Answer 1

Reset to default
4
$\begingroup$

This sounds like a problem of feature selection, if this is the case, I think you want to compute the mutual information between all subsets of features and the classification output. The subset with the highest mutual information will be the set of features that contains the most 'information' about the resulting classification of the record.

If you only have 3 features, you can compute all possible subsets in a reasonable amount of time, if your feature set grows larger, you'll have to approximate this (typically using a greedy approach: take feature with the highest MI at each step).

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ (+1) for mutual information. Additional remark: a) I suggest Information Gain as special case of mutual information. b) Automatic feature selection will not only remove the redundant but also all features which have a negative impact on class discrimination. $\endgroup$
    – steffen
    Commented Feb 14, 2011 at 10:14
  • $\begingroup$ Thanks! This sounds very promising, I will look into it. $\endgroup$
    – Björn Pollex
    Commented Feb 14, 2011 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.