Is spurious regression a problem for lasso and similar techniques?

Question

I was toying with R to see how the number of variables might affect spurious regression. Suppose that we have an $I(1)$ vector $y$ and a matrix $X$ with $I(1)$ columns. If the two are not related then OLS regression will be disastrous, with up to 50% of $X$'S columns showing significance. On the other hand suppose I set

$$y =X_1\beta + \epsilon $$

where $X_1$ is the first column of the $X$ matrix and $\epsilon$ is white noise. Then the regression works beautifully - the $y$ and $X_1$ form a cointegrating pair and the regression rightfully determines that the other columns are unrelated to the outcome, despite being nonstationary.

This begs the question - in situations where you have thousands or more variables and you would use regularized regression techniques, is spurious regression a problem? It seems that as long as there's at least one variable related to the outcome your regression will be fine.

The code for my experiment:

    nruns <- 1000
    nobs <- 1000
    nvars <- 100
    significant_coefs <- numeric(nruns)
    
    for(i in 1:nruns) {
      X <- replicate(nvars, cumsum(rnorm(nobs)))
      y <- X[, 1] + rnorm(nobs, sd = 1000)
      
      model <- lm(y ~ X)
      significant_coefs[i] <- sum(summary(model)$coefficients[, 4] <=
                                    0.05)
    }
    
    hist(significant_coefs)

To see the impact of spurious regression just change the $y$ variable to a random walk.

    nruns <- 1000
    nobs <- 1000
    nvars <- 100
    significant_coefs <- numeric(nruns)
    
    for(i in 1:nruns) {
      X <- replicate(nvars, cumsum(rnorm(nobs)))
      y <- cumsum(rnorm(nobs))
      
      model <- lm(y ~ X)
      significant_coefs[i] <- sum(summary(model)$coefficients[, 4] <= 
                               0.05)
    }
    
    hist(significant_coefs)

In the first case I get an average of 6 coefficients with p-values less than 0.05, in the second I get 51.

Answer 1

The reason that you get high a high amount of significant coefficients does not have to do with lasso but with the regression with time series. Time series often tend to correlate (more likely than two random Gaussian white noise vectors tend to correlate). See also this question: Why do these time series appear to be dependent? and this article written by Yule in 1926 "Why do we Sometimes get Nonsense-Correlations between Time-Series?--A Study in Sampling and the Nature of Time-Series"

The reasons are because your linear model is not a correct model for the error distribution.

What you have here are time series with autocorrelation. The consequence is that if a particular vector $x_i(t)$ has at some time $t$ a value close to $y(t)$ then it will also have values close at times nearby $t$. Effectively the degrees of freedom is not nobs.

The reason that it 'works' when you base $Y$ on a variable $X_1$ in the regressor matrix, is because now the autocorrelated part is fully explained by $X_1$ and the other variables are only used to explain the remainder which is Gaussian white noise instead of correlated noise.

---

Sidenote: You have 'spurious relation' and 'spurious correlation'. The latter term is a bit unclear or at least less common. You might want to explain this when you use it. See also Misunderstandings of "spurious correlation"?

Answer 2

You are mixing three different things: hypothesis testing, regularization, and causal inference. Spurious relationship (like correlation) is a relationship between variables that do exist in data but is not causal. It is not something that you can detect using $p$-values, because they serve merely for hypothesis testing, while causal inference is much more complicated than this. Life would be very easy if regularization would exclude the features that are not causally related to the target variable, but it doesn't work like this, as regularization works only on correlative relationships.

So first of all, you need to decide what you want to do:

• Test the hypotheses about the parameters, so test what the parameters are.
• Shrink the parameters using regularization, so force the parameters to be smaller.
• Seek causal relationships between the variables to explain the observed phenomenon.

Answer 3

In the scenario you described, you would usually use cross-validation to tune the regularization parameter of the regression. Cross-validation will tell you if the relationships you identified were spurious, since your model would have poor performance on the validation sets (but with potentially high variance). In that sense, spurious regression will not be a problem since you will know that your model has no predictive power.

However, performing linear regression on non-iid data is a bad idea. Your model will likely pick up spurious correlations, even though you will know that they are spurious through cross-validation. You should transform the data to stationary before performing the regression. This will allow the model to ignore irrelevant variables with a similar trend to your output variable.