# The disadvantage of using F-score in feature selection

F-score can be used to measure the discrimination of two sets of real-numbers and can be used for feature selection. However, I once read that

A disadvantage of F-score is that it does not reveal mutual information among features.

How to understand this statement, or why F-score has this kind of disadvantage.

• "... the discrimination of two sets of real-numbers"? What does that mean? – jbowman Dec 28 '11 at 22:03
• "F-score" can plausibly mean several different things in this context (Google it). To make this question less ambiguous, please describe what you mean by "F-score," or at least provide a link to a description. – whuber Dec 28 '11 at 22:29

Three years late, but it might help other people.

I guess you refer to F-score used in the paper of Chen and Lin (2006) : "Combining SVMs with Various Feature Selection Strategies". They use an example to explain what you ask :

I quote their words : "Both features of this data have low F-scores as the denominator (the sum of variances of the positive and negative sets) is much larger than the numerator."

In other words, F-score reveals the discriminative power of each feature independently from others. One score is computed for the first feature, and another score is computed for the second feature. But it does not indicate anything on the combination of both features (mutual information). This is the main weakness of F-score.

• (+1) Welcome to CV! Maybe you could provide additional information on how F-score is computed? This would make this answer more informative. Btw, I added a link to the paper you quote. – Tim Jan 13 '15 at 13:38

The F-score is a ratio of two variables: F = F1/F2, where F1 is the variability between groups and F2 is the variability within each group. In other words, a high F value (leading to a significant p-value depending on your alpha) means that at least one of your groups is significantly different from the rest, but it doesn't tell you which group.

Typically you select features that return high F-values and use those for further analysis.

In this reply, I assume that the questioned F-score is the one described in the article pointed out by @Guillaume Sutra. Here is the page describing the F-score, including its definition:

Let us first look at the intuition behind the F-score for feature selection. For simplicity, let us consider a binary classification problem (each sample in the dataset has one of two classes). Assume we have a large dataset on the format:

x1  x2  x3  ... class
0.3 0.5 0.1 ... A
0.1 0.7 0.4 ... B
0.1 0.1 0.2 ... A
0.2 0.4 0.2 ... A
0.5 0.7 0.8 ... B
... ... ... ... ...


The F-score is a uni-variate feature selection method, which means that it scores each of the features (x1, x2, x3, ...) individually (higher score is better), without considering that a feature may improve in combination with another feature. For example, let us assume that we want to score the feature x2. With respect to the x2 feature, assume that our dataset looks as follows:

If we spend 10 seconds drawing a normal distribution for both classes in Microsoft Paint, it looks as follows:

Now, if we want to predict the class of a datapoint only based on the x2 feature, the fact that the two normal distributions overlap makes it harder for us to make a good predictive model. Consider for instance these two extreme cases:

It is easy to see how the case to the left is much easier to predict. Notice that the overlap is reduced when

• The means of A and B are more separated (the numerator in the F-score definition)
• The variances of A and B are small (the denominator in the F-score definition)

The F-score captures these two properties, such that a high F-score reflects a small overlap.

Regarding your question about the F-score not revealing mutual information, I have made this example, inspired by the article:

For each of the features x1 and x2, the F-score is low because of the high variance. However, by combining the two features, you can perfectly separate the two classes A and B. Unfortunately, the F-score does not consider this.