# Is spurious regression a problem for lasso and similar techniques?

I was toying with R to see how the number of variables might affect spurious regression. Suppose that we have an $$I(1)$$ vector $$y$$ and a matrix $$X$$ with $$I(1)$$ columns. If the two are not related then OLS regression will be disastrous, with up to 50% of $$X$$'S columns showing significance. On the other hand suppose I set

$$y =X_1\beta + \epsilon$$

where $$X_1$$ is the first column of the $$X$$ matrix and $$\epsilon$$ is white noise. Then the regression works beautifully - the $$y$$ and $$X_1$$ form a cointegrating pair and the regression rightfully determines that the other columns are unrelated to the outcome, despite being nonstationary.

This begs the question - in situations where you have thousands or more variables and you would use regularized regression techniques, is spurious regression a problem? It seems that as long as there's at least one variable related to the outcome your regression will be fine.

The code for my experiment:

    nruns <- 1000
nobs <- 1000
nvars <- 100
significant_coefs <- numeric(nruns)

for(i in 1:nruns) {
X <- replicate(nvars, cumsum(rnorm(nobs)))
y <- X[, 1] + rnorm(nobs, sd = 1000)

model <- lm(y ~ X)
significant_coefs[i] <- sum(summary(model)$coefficients[, 4] <= 0.05) } hist(significant_coefs)  To see the impact of spurious regression just change the$y$variable to a random walk.  nruns <- 1000 nobs <- 1000 nvars <- 100 significant_coefs <- numeric(nruns) for(i in 1:nruns) { X <- replicate(nvars, cumsum(rnorm(nobs))) y <- cumsum(rnorm(nobs)) model <- lm(y ~ X) significant_coefs[i] <- sum(summary(model)$coefficients[, 4] <=
0.05)
}

hist(significant_coefs)


In the first case I get an average of 6 coefficients with p-values less than 0.05, in the second I get 51.

• Can you define "spurious regression" in your question? I don't know if it a common term, but I don't know it at least. Jun 8, 2018 at 15:30
• It's when things that aren't meaningful appear to be so. I'll add code so you can see what I mean. Jun 8, 2018 at 15:31
• @badmax "spurious" refers to the conclusions drawn from a regression model. Jun 8, 2018 at 15:46
• Are you trying to build a predictive model, or are you trying to identify which input variables the output variable is dependent on? Also, did I misunderstand your code, or are you running regressions on non-stationary data? In that case the significance test for the coefficients does not apply. Jun 8, 2018 at 15:48
• I am trying to build a predictive model while identifying the dependence structure. I am running regressions on non-stationary data to see what would happen. Jun 8, 2018 at 15:54

The reason that you get high a high amount of significant coefficients does not have to do with lasso but with the regression with time series. Time series often tend to correlate (more likely than two random Gaussian white noise vectors tend to correlate). See also this question: Why do these time series appear to be dependent? and this article written by Yule in 1926 "Why do we Sometimes get Nonsense-Correlations between Time-Series?--A Study in Sampling and the Nature of Time-Series"

The reasons are because your linear model is not a correct model for the error distribution.

What you have here are time series with autocorrelation. The consequence is that if a particular vector $$x_i(t)$$ has at some time $$t$$ a value close to $$y(t)$$ then it will also have values close at times nearby $$t$$. Effectively the degrees of freedom is not nobs.

The reason that it 'works' when you base $$Y$$ on a variable $$X_1$$ in the regressor matrix, is because now the autocorrelated part is fully explained by $$X_1$$ and the other variables are only used to explain the remainder which is Gaussian white noise instead of correlated noise.

Sidenote: You have 'spurious relation' and 'spurious correlation'. The latter term is a bit unclear or at least less common. You might want to explain this when you use it. See also Misunderstandings of "spurious correlation"?

You are mixing three different things: hypothesis testing, regularization, and causal inference. Spurious relationship (like correlation) is a relationship between variables that do exist in data but is not causal. It is not something that you can detect using $$p$$-values, because they serve merely for hypothesis testing, while causal inference is much more complicated than this. Life would be very easy if regularization would exclude the features that are not causally related to the target variable, but it doesn't work like this, as regularization works only on correlative relationships.

So first of all, you need to decide what you want to do:

• Test the hypotheses about the parameters, so test what the parameters are.
• Shrink the parameters using regularization, so force the parameters to be smaller.
• Seek causal relationships between the variables to explain the observed phenomenon.

In the scenario you described, you would usually use cross-validation to tune the regularization parameter of the regression. Cross-validation will tell you if the relationships you identified were spurious, since your model would have poor performance on the validation sets (but with potentially high variance). In that sense, spurious regression will not be a problem since you will know that your model has no predictive power.

However, performing linear regression on non-iid data is a bad idea. Your model will likely pick up spurious correlations, even though you will know that they are spurious through cross-validation. You should transform the data to stationary before performing the regression. This will allow the model to ignore irrelevant variables with a similar trend to your output variable.

• couldn't we run a regression on non-stationary data, hoping that a complex model (e.g. a Machine Learning model) can catch some of the non-stationary trends, then check for validation metrics on the (transformed) stationary data? Sep 19, 2018 at 13:37