Does decreasing correlation between response and predictors increase lasso objective function?

Question

Does decreasing correlation between response $y$ and predictors $X$ in a linear model increase lasso objective function $\Vert y- X\beta \Vert _2 ^2 /n + \lambda\Vert\beta \Vert_1$?

Can it be shown mathematically for a fixed lambda or a lambda produced from CV?

Answer 1

Without any additional constraints, decreasing the correlation between predictors and response may increase or decrease the lasso objective function. Here are two examples showing opposite effects.

Initial conditions (case A)

Suppose we start with random variables $X$ and $Y$ (both scalar-valued) where:

$$Y = X \beta + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2)$$

for some non-zero slope $\beta$ and noise variance $\sigma^2$. Sample a large dataset, then use lasso to estimate the slope, with some penalty strength $\lambda$. Note the value of the objective function and its components: the squared error term and lasso penalty term.

Case B

Proceed as in case A, but reduce the correlation between $X$ and $Y$ by doubling the noise variance. Sample a large dataset and run lasso with the same $\lambda$. Because the underlying linear relationship between $X$ and $Y$ is the same as in case A, we should find the same slope, and the lasso penalty term will not change. But, because the noise variance has increased, the squared error term will be greater. Therefore, the lasso objective function will increase.

Case C

Proceed as in case A, but reduce the correlation between $X$ and $Y$ by setting the slope to zero; the correlation will therefore be zero. Sample a large dataset and run lasso with the same $\lambda$. Lasso should find a slope near the true value of zero, meaning the penalty term will be near zero as well (i.e. less than in case A). Because the noise variance is the same as in case A, the squared error term will not increase. It may actually decrease because the shrinkage applied by lasso in case A causes the slope to be underestimated, inflating the error beyond the noise variance. Whereas, in case C, lasso can find the optimal slope of zero without any penalty. Therefore, the lasso objective function will decrease.