I have a problem that I'm trying to figure out.

Problem: Suppose we estimate the regression coefficients in a linear regression model by minimizing

$\sum_{i=1}^n (y_i - \beta_0-\sum_{j=1}^p \beta_j x_{ij})^2$ subject to $\sum_{j=1}^p \lvert{\beta_j}\lvert \le s$

for a particular value of s. Indicate which is correct:

(a) As we increase s from 0, the training RSS will:

i. Increase initially, and then eventually start decreasing in an inverted U shape.

ii. Decrease initially, and then eventually start increasing in a U shape.

iii. Steadily increase.

iv. Steadily decrease.

v. Remain constant.

(b) Repeat (a) for test RSS.

I saw someone who wrote that for (a) RSS will steadily decrease and for (b) it will decrease initially then increase in a U shape.

I don't understand why this is though. My whole understanding of Lasso is that is seeks to decrease variance and increase bias. Less variability should lead to a lower RSS and MSE. And I understand that as we increase s, we "relax" the constraint on our prediction coefficients and thus they will eventually just get the Least Squares Estimator.

Sorry if this seems totally off. I'm trying to wrap my mind around this :/ Thank you in advance.

  • 1
    $\begingroup$ You are correct that as we increase $s$, LASSO becomes closer to OLS. Try to elaborate the phrases "decreases variance" and "lower RSS" - i.e., variance and RSS of what? $\endgroup$ – juod Sep 10 '18 at 2:38
  • $\begingroup$ Are you asking me to elaborate the question's phrases or my interpretation? I guess I have some conceptual misunderstanding. I'm not sure how testing and training RSS is affected though. Is this relating to overfitting? I.e., constraining too much leads to overfitting of training data? But how do we know how the test data will behave? $\endgroup$ – Nicklovn Sep 10 '18 at 16:53

As discussed in comments under the question, this problem is based on understanding two concepts:

  1. $s$ controls the regularization strength: $s=0$ enforces an intercept-only model, $s=+\infty$ permits unconstrained least squares
  2. Overfitting means low RSS on train data, but high RSS on new data

Some misunderstanding may arise from using the term "variance" loosely. LASSO (or any other regularization, for that matter) reduces the variance of the estimate of underlying function, $\hat{f}(x)$, taken over various samples of $x$. In other words, new training data will produce similar estimates.

Thus, for (a) we have that relaxing LASSO regularization leads to less shrinkage, larger coefficients, and smaller train-data RSS. At some point, overfitting kicks in, and while train-data RSS continues to increase, test-data RSS will start decreasing.

(For the boundary case when none of the $x$ are predictive and the true model is $y=\beta_0 + \epsilon$, the correct answer is (b) iv, as we start with the optimal model $\beta_j=0$ and any relaxation only introduces overfit.)

| cite | improve this answer | |
  • $\begingroup$ So does this imply that training and test RSS behave opposite to each other? Also, if $s=0$ is this still overfitting? Can't this be seen as the opposite since we now are left with just the intercept and no slope. Why wouldn't this have higher RSS than the Least Squares Estimator? So for a you say that training data RSS will decrease and then start to increase? Sorry for my confusion. This is a new concept for me. $\endgroup$ – Nicklovn Sep 11 '18 at 13:44
  • $\begingroup$ @Nicklovn overfitting by definition means having a good fit (RSS) in test data that cannot be replicated in new data. A regularized model cannot have higher training RSS than a simple least squares model, so it cannot be more overfit than OLS. Hope that answers some of the questions. $\endgroup$ – juod Sep 12 '18 at 4:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.