What does a p-value mean in the context of feature selection?

Question

I think I understand the basic definition of p-values in statistics, but I'm confused what it would mean in the context of feature selection. For example, in scikit-learn you can do feature selection with the SelectPercentile class and tell it to, say, only keep the top 30 percent of features.

The SelectPercentile class has two attributes: scores_ and pvalues_. I assume to get the top 30 percent of features it just ranks them by their scores and takes the top 30 percent. But what would a pvalue mean in this context?

For reference, I ran some of my data through SelectPercentile and I got the following:

Feature scores: 
array([   71.63040161,  5156.66259766,  1368.79492188,   805.26611328,
         788.79217529,   110.83755493,   705.46398926,   854.82958984], dtype=float32)
Feature pvalues: 
array([  2.97808976e-17,   0.00000000e+00,   0.00000000e+00,
         0.00000000e+00,   0.00000000e+00,   8.89241668e-26,
         0.00000000e+00,   0.00000000e+00], dtype=float32)

Answer 1

It seems like SelectPercentile is able to select features according to different metrics of quality for the features. The specific metric used is determined by the score_func argument, which according to the documentation must be:

Function taking two arrays X and y, and returning a pair of arrays (scores, pvalues) or a single array with scores. Default is f_classif (see below “See also”). The default function only works with classification tasks.

So by default it uses f_classif as a measure of features' quality. Looking at the documentation of this function you can see that it computes "the ANOVA F-value", so that is where your p-value comes from.