2
$\begingroup$

Forward stagewise regression is a simple model selection algorithm related to least angle regression and LASSO. (see e.g. the LARS paper) It repeats the following steps, initializing a predictor $\hat{\mu} = 0$ and a residual $r = y$:

  1. For each covariate $j$, let $c_j$ be the correlation between $x_j$ and $r$.
  2. Let $j^* = \arg\max_{j} |c_j|$ and let $\hat{\mu} \leftarrow \hat{\mu} + \epsilon\cdot\textrm{sign}(c_{j^*})\cdot x_{j^*}$
  3. Let $r \leftarrow r - \hat{\mu}$

My question is:

are there conditions under which forward stagewise regression can be shown to converge to the true model?

I'm looking for any analysis at all, and would be happy to see one even if it made strong and unrealistic assumptions about the data distribution and the true model.

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you elaborate on how this is to happen more precisely? AFAIK, LARS will produce one best model of any of the dimensionalities considered, so we need to agree on how to proceed to choose between the different best models. $\endgroup$
    – Christoph Hanck
    Dec 28, 2015 at 13:58
  • $\begingroup$ Suppose I tell you that the true model (from which the data was generated) has $\ell_1$ weight exactly $k$, so there is no need to select the best value for $k$. I'm not sure what you mean by "best'' model, but if you have any reference showing that something like LARS selects "best'' models I would be happy to see them. $\endgroup$
    – Aaron
    Dec 28, 2015 at 15:19

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.