16
$\begingroup$

I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around.

To keep things simple, I'm going to look at the minimization of $$ -l(\beta, X, Y) + n \lambda p(\beta) $$ for some reasonable penalty function $p$ (and $l$ the loglikelihood). In theorems regarding the asymptotic properties of the resulting estimator, typically a requirement is imposed on $\lambda$, or more precisely two requirements: an upper and lower bound on its behaviour for large $n$ (e.g. $\lambda \rightarrow 0$ and $\sqrt{n} \lambda \rightarrow \infty$ for $n \rightarrow \infty$. This is a requirement that shows up in papers by Fan en Li (SCAD), Zou (Adaptive Lasso) and some others.

My issue with this is that it is never specified how to impose such boundaries. In practice, you have a single dataset and try to find the best possible value for the tuning parameter $\lambda$, but of course in this case the sample size doesn't change and definitely is no approaching infinity.

My guess is that it means that your method to select the best value for $\lambda$ (e.g. crossvalidation, AIC or BIC or similar) should be such that the limiting behaviour is as required, but noone ever proves this, or at least I have not been able to find it.

So, in short: can any of you explain to me how to work with these requirements for $\lambda$, or point me to papers/books/.../suggest a simulation experiment/ whatever that makes these issues clear. I'm hoping to prove similar asymptotic properties in settings beyond maximum likelihood, but then I need to understand the state of the art to its fullest.

EDIT:

Reading, re-reading and re-re-reading some of the papers, I finally realised that the asymptotic properties (of interest to me: consistent model selection and by extension, the oracle properties) perhaps do not require the tuning parameter selection to uphold the limiting behaviour on the parameter itself. The theorems typically show that any $\lambda$ series that satisfies the limiting conditions will result in an estimator with the properties of interest.

As such, I just have to pick my $\lambda$ that performs best, and "virtually promise" if I were to redo the analysis on a larger/smaller dataset, that I would scale that $\lambda$ accordingly.

If this is correct, this only leaves me with my classical problem around crossvalidation: here, the effectiveness of the model is evaluated on (e.g.) 9/10 of the data. Even if I scale $\lambda$ the right way, what guarantees that whichever criterion I'm using, scales along with it? This appears to be less of a problem with other methods of choosing the tuning parameter. Can anybody shed some light on this (I'm still trying to get my head around @Stefan Wager's comment, so maybe it's in there already)?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Analyzing the convergence rate of smoothing parameters selected by cross-validation seems like a difficult task. For example, suppose we want to do kernel density estimation and pick the kernel bandwidth by cross validation. Then, the selected bandwidth converges to the optimal one at a rate of n^{1/10}! (jstor.org/stable/10.2307/2346138) Of course, this result is probably not directly relevant, but it seems to indicate that cross-validated parameters may have bizzarre asymptotic behavior. $\endgroup$
    – Stefan Wager
    Apr 29, 2013 at 7:10
  • 1
    $\begingroup$ Hi, Nick. Are you familiar with the paper W. Fu and K. Knight (2000), Asymptotics for lasso-type estimators, Ann. Stat., vol 28, no. 5, 1356-1378? This paper doesn't address your question head-on, but I mention it because it gives a simple framework to see what actually matters, while also allowing for considerable generality. They use a fixed sequence $\lambda_n$, but you can adapt their Theorem 1 to a sequence dependent on the data (e.g., via cross-validation) without too much difficulty. $\endgroup$
    – cardinal
    Apr 30, 2013 at 13:18
  • 2
    $\begingroup$ It may also help for you to specify which asymptotic properties you are interested in. For example, (as you are likely to know) there is a significant difference between asymptotic consistency of the parameters and asymptotic consistency of model selection. There are even some results to the effect of not being able to get both of them together! $\endgroup$
    – cardinal
    Apr 30, 2013 at 13:20
  • $\begingroup$ @cardinal: I'm looking for oracle properties (thus, at the least model selection consistency). The conflicting effect is indeed described in the adaptive lasso paper (Zou) for the lasso. I had the Fu & Knight paper at hand but for some reason it was not in my list of "active" papers, so I'll check it out again. $\endgroup$
    – Nick Sabbe
    May 2, 2013 at 7:31
  • $\begingroup$ Since the oracle is getting mentioned, here is a question on the definition of the oracle property. $\endgroup$
    – Richard Hardy
    Aug 19, 2016 at 6:48

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.