Using p-values and correlated variables

Question

Suppose I have the true model that $y= 1+3x_1+4x_2+\xi$ where $\xi\sim\mathcal{N}(0,1)$. However, $x_2=5x_1+\eta$ where $\eta\sim\mathcal{N}(0,\sigma^2)$.

If I was to regress $y$ on $x_1,x_2$ I use the following code in R:

x1=1:20
sigma=0.1
x2=x1+sigma*rnorm(20,1)
y=y=1+3*x1+4*x2+rnorm(20,1)
data=data.frame(y,x1,x2)
summary(lm(y~.,data))

For $\sigma=0.1$ I get the following output:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   2.3771     0.5735   4.145 0.000677 ***
x1            3.5004     2.3680   1.478 0.157643    
x2            3.4736     2.3700   1.466 0.161002    
---

However, if I change it to $\sigma=1$ I get:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   2.5992     0.4652   5.588 3.27e-05 ***
x1            3.0217     0.3171   9.529 3.13e-08 ***
x2            3.9469     0.2978  13.252 2.17e-10 ***
---

Now when I try graphical lasso (glasso) using R's huge library I get the following graph: (This part is just to illustrate that some algorithms can pick up the correlation. Not necessarily related to question below.)

library(huge)
a=huge(as.matrix(data),method="glasso")
g=graph.adjacency(a$path[[10]],mode="undirected")
plot(g)

The question is:

1. When I know that some input variables are linearly correlated what do the p-values mean? I'm asking this in a general context and not in relation to the demonstrated problem.

2. What if $x_2$ was $f(x_1)$ where $f(\cdot)$ is non-linear? What is the interpretation of a p-value?

3. (optional) Can graphical Lasso replace stepwise regression to some extent? I understand that this algorithm is highly dependent on the regularising parameter.

Answer 1

The core issue here goes by the name of collinearity. As $\sigma$ decreases toward zero, your $x_1$ and $x_2$ explanatory variables become essentially identical (that is, except for the scale factor 5). In the limiting case where $x_2 \equiv 5 x_1$, your given 'true' data-generating process (with $\beta_1=3,\, \beta_2=4$) is observationally equivalent to any DGP that satisfies the linear constraint $\beta_1 + 5 \beta_2 = 23$. (This phenomenon is also discussed more abstractly in terms of the matrix $X^T X$ becoming singular and therefore non-invertible.) That is to say, in the limit $\sigma \rightarrow 0$, there's only 1 linear constraint on the 2 coefficients $\beta_1$ and $\beta_2$. Two variable, one constraint; therefore no unique solution. So the confidence intervals on the $\beta$ estimates span $(-\infty, \infty)$; the p-values in that limit would be 1. In either case, the conclusion is that you can't know much about the coefficients individually. Merely one of the many things you can't know is whether these coefficients are 'significantly different from zero'. (Note that you can still know something about a linear combination of the parameters--in this case, about the quantity $\beta_1 + 5 \beta_2$.)

The case where $x_2 = f(x_1)$ is actually quite commonly encountered in the form of models containing linear and quadratic (or other higher-order) terms in an explanatory variable. In such cases, a common practice (which I'm not endorsing) may be to perform model selection in a nested fashion: the quadratic term would be tested first, and discarded if 'not significant'; if the quadratic term were retained, then so would be the linear term. If the quadratic term were discarded, the model might be re-estimated, and model selection done on the linear term. Of course, such model selection procedures undermine whatever meaning the reported p-values or other standard inferential statistics might be thought to have.

If your purpose in modeling is to learn something about nature (from the parameter estimates that emerge from analysis), then finding multicollinearity in your data just means you didn't collect data with enough natural variability to tell you about all of your parameters individually. If, on the other hand, your main purpose is prediction, then multicollinearity is less problematic. All you really need in that case is one of the class of observationally equivalent DGPs that includes the 'true' DGP.

I'm guessing from the final part of your question that you may be finding much of your motivation from conceiving of your modeling problem as one of variable selection. Frank Harrell has a great deal to say about this, especially about stepwise variable selection (bad) and some suitable alternatives. Please see his answer to this question to start, and also his new book.