12
$\begingroup$

Is comparing features using F-regression the same as correlating features with the label individually and observing the $R^2$ value?

I have often seen my colleagues use an F regression for feature selection in their machine learning pipeline from sklearn:

sklearn.feature_selection.SelectKBest(score_func=sklearn.feature_selection.f_regression...)`  

Some please tell me - why does it give the same results as just correlating it with the label/depedendent variable?

It is not clear to me the advantage of using F_regression in feature selection.

Here's my code: I'm using the mtcars dataset from R:

import pandas as pd
import numpy as np
from sklearn import feature_selection
from sklearn.linear_model import LinearRegression

#....load mtcars dataset into a pandas dataframe called "df", not shown here for conciseness

# only using these numerical columns as features ['mpg', 'disp', 'drat', 'wt']
# using this column as the label:  ['qsec']

model = feature_selection.SelectKBest(score_func=feature_selection.f_regression,\
                                      k=4)

results = model.fit(df[columns], df['qsec'])

print results.scores_
print results.pvalues_

# Using just correlation coefficient:

columns = ['mpg', 'disp', 'drat', 'wt']
for col in columns:
    lm = LinearRegression(fit_intercept=True)
    lm.fit(df[[col]], df['qsec'])
    print lm.score(df[[col]], df['qsec'])

As suspected, the ranking of the features is exactly the same:

scores using f_regression:

[ 6.376702    6.95008354  0.25164249  0.94460378]


 scores using coefficient of determination:

0.175296320261  
0.18809385182
0.00831830818303
0.0305256382746

As you can see, the second feature is ranked the highest, the first feature is second, the fourth feature is third, and the third feature is last, in both cases.

Is there ever a case where the F_regression would give different results, or would rank the features differently in some way?

EDIT: To summarize, I'd like to know if these two rankings of features ever give different results:

1) ranking features by their F-statistic when regressing them with the outcome individually (this is what sklearn does) AND,

2) ranking features by their R-squared value when regressing them with the outcome , again individually.

$\endgroup$
  • $\begingroup$ SO went down immediately after I posted this, which I'm sure hurt the chances of it getting any attention. $\endgroup$ – Hunle Mar 30 '16 at 15:55
  • 1
    $\begingroup$ Your question contains the term "F-regression". What's that and how is it different from regression? ... (Edit:) Something occurs to me just now: are you referring to an F-test (or perhaps just an F-statistic) for the overall regression against a nil-null (i.e. intercept only)? $\endgroup$ – Glen_b Apr 15 '16 at 1:01
  • $\begingroup$ I'm referring to the F-test. In regression, the F-test and hence F-statistic, is used to test the null hypothesis that there is no relationship between the regressor and the outcome/label. sklearn refers to it as F-regression, which is perhaps a bit misleading since it's actually a test. scikit-learn.org/stable/modules/generated/… $\endgroup$ – Hunle Apr 15 '16 at 6:36
  • $\begingroup$ Your comment there suggests you only have one regressor variable (in which case why are you talking about feature selection?) $\endgroup$ – Glen_b Apr 15 '16 at 6:38
  • 2
    $\begingroup$ Could you please edit that explanation into your question? $\endgroup$ – Glen_b Apr 15 '16 at 6:47
11
+50
$\begingroup$

TL:DR

There won't be a difference if F-regression just computes the F statistic and pick the best features. There might be a difference in the ranking, assuming F-regression does the following:

  • Start with a constant model, $M_0$
  • Try all models $M_1$ consisting of just one feature and pick the best according to the F statistic
  • Try all models $M_2$ consisting of $M_1$ plus one other feature and pick the best ...

As the correlation will not be the same at each iteration. But you can still get this ranking by just computing the correlation at each step, so why does F-regression takes an additional step? It does two things:

  • Feature selection: If you want to select the $k$ best features in a Machine learning pipeline, where you only care about accuracy and have measures to adjust under/overfitting, you might only care about the ranking and the additional computation is not useful.
  • Test for significance: If you are trying to understand the effect of some variables on an output in a study, you might want to build a linear model, and only include the variables that are significantly improving your model, with respect to some $p$-value. Here, F-regression comes in handy.

What is a F-test

A F-test (Wikipedia) is a way of comparing the significance of the improvement of a model, with respect to the addition of new variables. You can use it when have a basic model $M_0$ and a more complicated model $M_1$, which contains all variables from $M_0$ and some more. The F-test tells you if $M_1$ is significantly better than $M_0$, with respect to a $p$-value.

To do so, it uses the residual sum of squares as an error measure, and compares the reduction in error with the number of variables added, and the number of observation (more details on Wikipedia). Adding variables, even if they are completely random, is expected to always help the model achieve lower error by adding another dimension. The goal is to figure out if the new features are really helpful or if they are random numbers but still help the model because they add a dimension.


What does f_regression do

Note that I am not familiar with the Scikit learn implementation, but lets try to figure out what f_regression is doing. The documentation states that the procedure is sequential. If the word sequential means the same as in other statistical packages, such as Matlab Sequential Feature Selection, here is how I would expect it to proceed:

  • Start with a constant model, $M_0$
  • Try all models $M_1$ consisting of just one feature and pick the best according to the F statistic
  • Try all models $M_2$ consisting of $M_1$ plus one other feature and pick the best ...

For now, I think it is a close enough approximation to answer your question; is there a difference between the ranking of f_regression and ranking by correlation.

If you were to start with constant model $M_0$ and try to find the best model with only one feature, $M_1$, you will select the same feature whether you use f_regression or your correlation based approach, as they are both a measure of linear dependency. But if you were to go from $M_0$ to $M_1$ and then to $M_2$, there would be a difference in your scoring.

Assume you have three features, $x_1, x_2, x_3$, where both $x_1$ and $x_2$ are highly correlated with the output $y$, but also highly correlated with each other, while $x_3$ is only midly correlated with $y$. Your method of scoring would assign the best scores to $x_1$ and $x_2$, but the sequential method might not. In the first round, it would pick the best feature, say $x_1$, to create $M_1$. Then, it would evaluate both $x_2$ and $x_3$ for $M_2$. As $x_2$ is highly correlated with an already selected feature, most of the information it contains is already incorporated into the model, and therefore the procedure might select $x_3$. While it is less correlated to $y$, it is more correlated to the residuals, the part that $x_1$ does not already explain, than $x_2$. This is how the two procedure you propose are different.

You can still emulate the same effect with your idea by building your model sequentially and measuring the difference in gain for each additional feature instead of comparing them to the constant model $M_0$ as you are doing now. The result would not be different from the f_regression results. The reason for this function to exists is to provide this sequential feature selection, and additionnaly converts the result to an F measure which you can use to judge significance.


The goal of the F-test is to provide significance level. If you want to make sure the features your are including are significant with respect to your $p$-value, you use an F-test. If you just want to include the $k$ best features, you can use the correlation only.


Additional material: Here is an introduction to the F-test you might find helpful

$\endgroup$
  • $\begingroup$ OK, now I see how this method of feature selection can guard against multicollinearity. I suppose if I'm running something like a Random Forest, which is not as susceptible to multicollinearity, then this feature selection method might not be as applicable. thank you @Winks $\endgroup$ – Hunle Apr 16 '16 at 18:31
  • $\begingroup$ Beware of using correlation only as a measure of feature importance. It measures linear dependance between variables, and tells you a feature (might be) is good for a linear model. This is not an assumption you can make for random forest, as trees can learn much more than linear relations. Correlation is not all there is (see Anscombe Dataset (Wikipedia). $\endgroup$ – Winks Apr 16 '16 at 18:58
  • $\begingroup$ What is the "slight problem with p-values" that you refer to? And, is there an issue of multiple comparisons since we're testing on the same data each time? $\endgroup$ – Hunle May 22 '16 at 6:26
  • $\begingroup$ One last thing (sorry to be annoying). As you say "...and then to $M_2$ there would be a difference in your scoring". For this intermediate step, the $R^2$ value would always be monotonically related to the p-value in an F distribution, since N and k are fixed in computing the F-statistic (stats.stackexchange.com/questions/50425/…). So, I guess I don't understand the need for the the F-test or F distribution. Do you? $\endgroup$ – Hunle May 22 '16 at 20:31
  • 1
    $\begingroup$ @Hunle I added the reason to use an F-test at the end of my answer. Basically, it's if you care about significance. Concerning the slight problem with $p$-values, I think you are on the point. It feels to me as we are testing the same thing multiple times (obligatory XKCD). However, since the F-test is used in practice, it might 1) not do what I think it does 2) not be a problem since we're not testing the exactly same thing at each step. $\endgroup$ – Winks May 23 '16 at 7:03
15
$\begingroup$

I spent some time looking through the Scikit source code in order to understand what f_regression does, and I would like to post my observations here.

The original question was:

Q: Does SelectKBest(f_regression, k = 4) produce the same result as using LinearRegression(fit_intercept=True) and choosing the first 4 features with the highest scores?

The answer is yes. Moreover, the relative ordering given by the scores is the same.

Here is what f_regression does, on input matrix $X$ and array $y$. For every feature $X[:, i]$ it computes the correlation with $y$: $$ \rho_i = \frac{(X[:, i] - mean(X[:, i])) * (y - mean(y))}{std(X[:, i]) * std(y)}. $$ Then it computes the F-statistic $$ F_i = \frac{\rho_i^2}{1 - \rho_i^2}*(n-2), $$ where $n = len(y)$, the number of samples (there is a slight difference if parameter center is False; then it multiplies with $n-1$). These F-values are then returned, together with the associated p-values. So the result is a tuple (F-values, p-values). Then SelectKBest takes the first component of this tuple (these will be the scores), sorts it, and picks the first $k$ features of $X$ with the highest scores. There is no sequential application or anything, and the p-values are not used either.

Now let $R_i^2$ be the score computed by LinearRegression for $X[:, i]$ and $y$. This is a regression on a single variable, so $R_i^2 = \rho_i^2$. Then $$ R_i^2 < R_j^2 \Leftrightarrow \frac{\rho_i^2}{1 - \rho_i^2} < \frac{\rho_j^2}{1 - \rho_j^2} \Leftrightarrow F_i < F_j. $$ Hence there is no difference between f_regression and LinearRegression. Although one could construct a model sequentially, this is not what SelectKBest does.

$\endgroup$
  • 2
    $\begingroup$ Wow, so` SelectKBest` does not construct a model sequentially. $\endgroup$ – Hunle Feb 6 '17 at 16:04
  • $\begingroup$ For what it's worth, I agree with user43451's interpretation. And, I wish sklearn would just call it a correlation ranking of single features. F-test, for me, bring in the notion of sequential models as Winks alluded to in his accepted answer. $\endgroup$ – MrDrFenner May 25 '18 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.