Forward stagewise regression is a simple model selection algorithm related to least angle regression and LASSO. (see e.g. the LARS paper) It repeats the following steps, initializing a predictor $\hat{\mu} = 0$ and a residual $r = y$:

  1. For each covariate $j$, let $c_j$ be the correlation between $x_j$ and $r$.
  2. Let $j^* = \arg\max_{j} |c_j|$ and let $\hat{\mu} \leftarrow \hat{\mu} + \epsilon\cdot\textrm{sign}(c_{j^*})\cdot x_{j^*}$
  3. Let $r \leftarrow r - \hat{\mu}$

My question is:

are there conditions under which forward stagewise regression can be shown to converge to the true model?

I'm looking for any analysis at all, and would be happy to see one even if it made strong and unrealistic assumptions about the data distribution and the true model.

  • $\begingroup$ Could you elaborate on how this is to happen more precisely? AFAIK, LARS will produce one best model of any of the dimensionalities considered, so we need to agree on how to proceed to choose between the different best models. $\endgroup$ – Christoph Hanck Dec 28 '15 at 13:58
  • $\begingroup$ Suppose I tell you that the true model (from which the data was generated) has $\ell_1$ weight exactly $k$, so there is no need to select the best value for $k$. I'm not sure what you mean by "best'' model, but if you have any reference showing that something like LARS selects "best'' models I would be happy to see them. $\endgroup$ – Aaron Dec 28 '15 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.