Lasso: Effect of $l_1$ Constraint on $RSS$? I have a problem that I'm trying to figure out.
Problem: Suppose we estimate the regression coefficients in a linear regression model by minimizing 
$\sum_{i=1}^n (y_i - \beta_0-\sum_{j=1}^p \beta_j x_{ij})^2$ subject to $\sum_{j=1}^p \lvert{\beta_j}\lvert \le s$
for a particular value of s. Indicate which is correct:
(a) As we increase s from 0, the training RSS will:
i. Increase initially, and then eventually start decreasing in an inverted U shape.
ii. Decrease initially, and then eventually start increasing in a U shape.
iii. Steadily increase.
iv. Steadily decrease.
v. Remain constant.
(b) Repeat (a) for test RSS.

I saw someone who wrote that for (a) RSS will steadily decrease and for (b) it will decrease initially then increase in a U shape.
I don't understand why this is though. My whole understanding of Lasso is that is seeks to decrease variance and increase bias. Less variability should lead to a lower RSS and MSE. And I understand that as we increase s, we "relax" the constraint on our prediction coefficients and thus they will eventually just get the Least Squares Estimator.
Sorry if this seems totally off. I'm trying to wrap my mind around this :/ Thank you in advance.
 A: As discussed in comments under the question, this problem is based on understanding two concepts:  


*

*$s$ controls the regularization strength: $s=0$ enforces an intercept-only model, $s=+\infty$ permits unconstrained least squares

*Overfitting means low RSS on train data, but high RSS on new data


Some misunderstanding may arise from using the term "variance" loosely. LASSO (or any other regularization, for that matter) reduces the variance of the estimate of underlying function, $\hat{f}(x)$, taken over various samples of $x$. In other words, new training data will produce similar estimates.
Thus, for (a) we have that relaxing LASSO regularization leads to less shrinkage, larger coefficients, and smaller train-data RSS. At some point, overfitting kicks in, and while train-data RSS continues to increase, test-data RSS will start decreasing.
(For the boundary case when none of the $x$ are predictive and the true model is $y=\beta_0 + \epsilon$, the correct answer is (b) iv, as we start with the optimal model $\beta_j=0$ and any relaxation only introduces overfit.)
