Regularization in forward stagewise algorithm of LASSO regression

Question

I am trying to implement the LASSO regression from scratch to better understand it. For now I just follow the pseudocode from here (page 5) for forward stagewise. It goes like this:

1. Init with $r=r-\bar y$, $\beta_i = 0$
2. Find the predictor $x_j$ most correlated with $r$.
3. Update $beta_j \leftarrow \beta_j + \delta_j$, where $\delta_j = \epsilon \cdot sign(corr(r, x_j))$
4. Update $r \leftarrow r - \delta_j x_j$
5. Repeat 2–4 until no predictor correlates with $r$.

Where should regularization parameter appear in this? When defining LASSO, we want $\sum\beta_i\le C$, but I don't see at which point of the algorithm we regularize. Am I confusing something?

Also, is it possible to not fit intercept with this algorithm? Usually we add a column $x_0$ with constants for that, but with this algorithm it will always produce the correlation of 0. So is the data normalization (i.e., bring all means to 0) the only way?

Answer 1

At each iteration, the solution is optimal (in a sense) for that particular value of $\Sigma |\beta|=C$. The algorithm produces a path that gives solutions for various values of $C$. Note that the path this algorithm produces is somewhat different than the LASSO path.

Regarding the second question, there is nothing stopping one from initializing with $r=y$ and including an intercept as a predictor (or not). In general, it just doesn't usually make sense to regularize the intercept term (or make an analysis that isn't invariant to shifts in $y$).