I have a general question about regularization in linear models. This was inspired by Knight and Fu's 2000 paper Asymptotics for Lasso-Type Estimators. Their basic set-up is that we have a fixed number $p$ of variables, and $n$ observations on them with $n \to \infty$. For each $n$ let $C_n = \frac 1n X_n^T X_n$, and assume that $C_n \to C$ where $C$ is positive semi-definite. The authors also assume that $\frac 1n \max_{i \leq n} x_i^T x_i \to 0$, where $x_i$ is a single observation on our $p$ variables.

For each $n$ we use regularization parameter $\lambda_n$, so that $$ \hat \beta_n = \text{argmin}_\phi \vert \vert Y_n - X_n \phi \vert \vert_2^2 + \lambda_n \vert \vert \phi \vert \vert_\gamma^\gamma $$ for $\gamma > 0$.

They consider cases where $\lambda_n / n^\alpha \to \lambda_0 \geq 0$ for typically $\alpha = \frac 12$ or $\alpha=1$, but their results are only interesting if $\lambda_0 > 0$ since otherwise it's just OLS (in the limit, but that's the case we're considering).

My question: in practice, I've never encountered a situation where I need more regularization with more data. To me the most reasonable thing is that $\lambda_n = o(1)$ so that asymptotically we get consistency and the standard CLT and all that other good stuff. The only exception I can think of is if $C_n \to C$ with $C$ singular, but that seems like a rather special case and the authors of this paper don't focus on it. Other than mathematical curiosity, what's the utility of results like these where the amount of regularization increases with more data?

  • 1
    $\begingroup$ this seems to just be since they don't scale the squared error, which grows like $\mathcal{O}(n)$. $\endgroup$ – user795305 Jun 1 '17 at 13:04
  • $\begingroup$ @Ben good point, the value of the penalty will be completely dominated by the loss if $\lambda_n$ doesn't grow. But i guess my question is why is that so bad? Of course we won't get to prove theorems like those in the paper, but once $n \gg p$, like say $p=10^3$ and $n=10^{10}$, by the consistency of $\hat \beta$ we'll probably be so close to $\beta$ that not regularizing is quite reasonable. Maybe this is to account for cases where we're getting more "adversarial" data and we do still need to penalize? $\endgroup$ – alfalfa Jun 8 '17 at 16:44
  • $\begingroup$ when $\alpha = 0.5$, the (appropriately scaled) tuning parameter $\tilde{\lambda_n} = n^{-1}\lambda_n$ goes to zero, and when $\alpha = 1$, the tuning parameter $\tilde{\lambda}_n$ is roughly constant. Typical rates of $\tilde\lambda$ are $\sqrt{\frac{\log p}{n}}$, which does goes to zero under high dimensional rates. So you're intuition about the tuning parameter is right. Also, on the last part of your comment, in these theorems people assume gaussian (or subgaussian) error, so the data isn't adversarial. $\endgroup$ – user795305 Jun 8 '17 at 17:28
  • $\begingroup$ @Ben thanks for the clarification, that makes a lot of sense! If you flesh that out into an answer I'd be happy to accept it. Re: the adversarial thing, I was thinking of the new observations $(\vec x, y)$ that we're getting as $n \to \infty$, not just the new errors, but now I don't think that matters much either way $\endgroup$ – alfalfa Jun 20 '17 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.