Is it normal that p-value getting from F-regression is very small (even zero)?

Question

I am trying to evaluate the significance of two attributes: article length and article topic. The target variable is total article reading time (in seconds). The topic attribute is categorical. So I convert it to dummy variable. The sample training data is as below:

Y:74    X:1829  0   0   1   0   0   0   0   0   0   0
Y:86    X:2739  0   0   0   0   0   0   0   0   0   1

The first feature of X is article length.

Each article must belong to only one topic category. I have filter out the topic category that has less than 100 articles. In total, 100000 articles are in the training data.

I use f_regression function of scikit-learn to get p-values of the features.

vec = DictVectorizer()
sparse_X = vec.fit_transform(clean_features)
F, pval = f_regression(sparse_X.toarray(), numpy.array(clean_targets))

print(F)
print(pval)
print()
print(list(zip(vec.feature_names_, pval)))

The output is as below:

# F score:
[  1.72954024e+01   1.01292804e+00   1.64323868e+01   1.07647829e+02
   1.19945277e+01   5.17789209e+01   7.45631971e+01   2.78030969e+01
   1.04509561e+02   1.01995319e-03   8.03501196e+01]

# p-values:
[  3.20479428e-05   3.14207137e-01   5.04906967e-05   3.40013618e-25
   5.34004824e-04   6.29625311e-13   6.03819906e-18   1.34853446e-07
   1.65140109e-24   9.74522651e-01   3.23873092e-19]

[('body_length', 3.2047942838924331e-05), ('channel=asia', 0.31420713732034955), 
('channel=business', 5.0490696668669388e-05), ('channel=entrepreneurs', 3.4001361796454458e-25), 
('channel=investing', 0.00053400482413538117), ('channel=leadership', 6.2962531135432702e-13), 
('channel=lifestyle', 6.0381990610318036e-18), ('channel=lists', 1.3485344572490546e-07), 
('channel=opinions', 1.651401087421023e-24), ('channel=personalfinance', 0.97452265059722243), 
('channel=technology', 3.238730922292655e-19)]

I find that the p-values of most features are extremely small.

I also see that the more training data I use, the less p-values. When I include 2,000,000 training data. The p-values of some features are even zero.

Is it normal? Am I correct here?

Answer 1

If the effects are not null (i.e. there's some degree of linear relationship with the features, even if it's very small), then yes, the p-values will tend to decrease as the sample size increases.

Consider this: the standard errors of the estimates decrease as the square root of $n$, so as you add more data (let's assume the values are added in random order and that you start with a reasonable number of observations), the estimates will tend to stay more or less in about the same place but the standard errors of those estimates will get smaller, making them more standard errors from 0. The coefficient F-statistics are the squares of the number of standard errors from 0, so these will get larger and larger -- you will tend to be further out into the tail of the null distribution and so get smaller p-values.

The correct p-values will not be exactly zero but they can be extremely small.