How should I interpret the p-values (i.e. t-tests) in regressions, and can I use them for feature selection?

Question

I'm trying to do an OLS regression with several independent variables, and want to better understand how to interpret the p-values from doing the t-tests on the independent variables within my regression. For example, here is my result:

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.612
Model:                            OLS   Adj. R-squared:                  0.497
Method:                 Least Squares   F-statistic:                     5.353
Date:                Fri, 11 Jan 2013   Prob (F-statistic):            0.00390
Time:                        16:12:03   Log-Likelihood:                -239.61
No. Observations:                  23   AIC:                             491.2
Df Residuals:                      17   BIC:                             498.0
Df Model:                           5                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
const       4.268e+05   1.85e+04     23.092      0.000      3.88e+05  4.66e+05
x1           -70.4536   2230.755     -0.032      0.975     -4776.936  4636.028
x2         -2.384e+04   1.25e+04     -1.905      0.074     -5.02e+04  2565.514
x3         -3821.8439   3848.891     -0.993      0.335     -1.19e+04  4298.607
x4          4030.8183   2295.228      1.756      0.097      -811.689  8873.325
x5         -3.955e+04   1.73e+04     -2.282      0.036     -7.61e+04 -2977.451
==============================================================================
Omnibus:                        2.870   Durbin-Watson:                   1.674
Prob(Omnibus):                  0.238   Jarque-Bera (JB):                1.326
Skew:                          -0.227   Prob(JB):                        0.515
Kurtosis:                       4.085   Cond. No.                         21.8
==============================================================================

From what I understand, if the p-values are above a certain threshold for a given variable (e.g. p-value > 0.05) as is the case with variable x1's pvalue=0.975, then one can say that this particular regression doesn't gain any additional information from having this variable in there. If I'm misunderstanding or generalizing too much, let me know.

What else is confusing me is that same variable, x1, when I run a regression with just x1 and x5, x1's p-value=0.05. I'm guessing that I interpret this as, x1 has some useful information, but when compared with the information carried by x2, x2 and x4 together, x1 isn't useful.

Regarding feature selection, would it be correct to try all various subsets of x1 through x5, throw out those that contain an independent variable whose p-value > 0.05, and then use the remaining combinations with cross-validation to find the best model parameters?

My end goal is to do feature selection from a large set of variables, and maybe p-values are not the best thing to use for this. In either case, I would like to better understand these p-values, and if you have a favorite feature selection method, I'd love to hear as well. Thanks

Answer 1

You write

From what I understand, if the p-values are above a certain threshold for a given variable (e.g. p-value > 0.05) as is the case with variable x1's pvalue=0.975, then one can say that this particular regression doesn't gain any additional information from having this variable in there.

This is not correct. Every variable adds some information, unless it's just random noise.

A p-value has a very specific meaning:

If, in the population from which this sample was randomly drawn, the effect size associated with this p-value was 0, what would be the probability that, in a sample of this size, we would get a test statistic this far from 0 or farther?

You also write:

What else is confusing me is that same variable, x1, when I run a regression with just x1 and x5, x1's p-value=0.05. I'm guessing that I interpret this as, x1 has some useful information, but when compared with the information carried by x2, x2 and x4 together, x1 isn't useful.

Rather than looking at p-values, I would look at effect sizes and changes in them.

In addition, the phrase "when compare with" is not right. When you have multiple indepdent variables the effects are each after controlling for the others; that is, holding the other variables in the regression constant, what are the effects of this variable?

Another thing I just noticed: You have only 23 observations!

You should be looking at a maximum of 2 variables at a time. The model that you present (with 5 variables) is almost surely overfit.

Answer 2

I would interpret those t statistics with caution. The beta coefficients in the regression model are calculated by taking all of the variables into account, and the t statistics are based on all of the variables, so it can be misleading to interpret them individually. Each variable's estimate is influenced by the presence of others, and can be masked by the other variables. For example, two variables could be non-significant (when viewed in the full model), but either one individually could be significant and important for the model.

Better would be to try a function that searches and tests for optimal subsets of regression variables - like regsubsets in R. In effect, what happens is that one tests a variety of models and tests them against each other using some criterion (like $R^2$ or AIC). The documentation of your variable hunting function/program should explain what procedure is being used.

Answer 3

Regarding feature selection: I would personally argue not to be based on p-values regarding feature selection. Use back-wards elimination; start with fitting a model with all the variables of interest and calculate your "favourite" information criterion (eg. AIC), then sequentially exclude one of your feature and recalculate the relevant AIC. Caveat: AIC is comparable among nested models.

Given your rather small dataset in the current case I might even consider BIC (which is somewhat more stringent than AIC). Additionally you might want to consider bootstrapping your sample and recalculating this procedure.

Regarding p-values, to quote Mackay (Chapt. 37) :

p-value is the probability, given a null hypothesis for the probability distribution of the data, that the outcome would be as extreme as, or more extreme than, the observed outcome.

Which is practically a (rather abstract) restatement of Peter Flom's answer. I would guess your initial statement would be valid if you said "enough information" instead of "any additional information". Adding features does decrease the degrees of freedom of your model.