In this reply, I assume that the questioned F-score is the one described in the article pointed out by @Guillaume Sutra. Here is the page describing the F-score, including its definition:
Let us first look at the intuition behind the F-score for feature selection. For simplicity, let us consider a binary classification problem (each sample in the dataset has one of two classes). Assume we have a large dataset on the format:
x1 x2 x3 ... class
0.3 0.5 0.1 ... A
0.1 0.7 0.4 ... B
0.1 0.1 0.2 ... A
0.2 0.4 0.2 ... A
0.5 0.7 0.8 ... B
... ... ... ... ...
The F-score is a uni-variate feature selection method, which means that it scores each of the features (x1, x2, x3, ...) individually (higher score is better), without considering that a feature may improve in combination with another feature. For example, let us assume that we want to score the feature x2. With respect to the x2 feature, assume that our dataset looks as follows:
If we spend 10 seconds drawing a normal distribution for both classes in Microsoft Paint, it looks as follows:
Now, if we want to predict the class of a datapoint only based on the x2 feature, the fact that the two normal distributions overlap makes it harder for us to make a good predictive model. Consider for instance these two extreme cases:
It is easy to see how the case to the left is much easier to predict. Notice that the overlap is reduced when
- The means of A and B are more separated (the numerator in the F-score definition)
- The variances of A and B are small (the denominator in the F-score definition)
The F-score captures these two properties, such that a high F-score reflects a small overlap.
Regarding your question about the F-score not revealing mutual information, I have made this example, inspired by the article:
For each of the features x1 and x2, the F-score is low because of the high variance. However, by combining the two features, you can perfectly separate the two classes A and B. Unfortunately, the F-score does not consider this.