10
$\begingroup$

There are many methods to perform regularization -- $L_0$, $L_1$, and $L_2$ norm based regularization for example. According to Friedman Hastie & Tibsharani, the best regularizer depends on the problem: namely the nature of the true target function, the particular basis used, signal-to-noise ratio, and sample size.

Is there any empirical research comparing methods and performance of various regularization methods?

$\endgroup$
9
  • 1
    $\begingroup$ All three authors are at Stanford. Why not just ask one of them directly. Rob Tibshirani is very approachable and so is Jerry Friedman. Friedman did a lot of original research in regularized regression. So he may be the better choice. $\endgroup$ – Michael R. Chernick May 15 '12 at 22:10
  • 1
    $\begingroup$ Of course I can't say that I gave him the answer. But directing him to the best person to answer the question seems like more than just an ordinary comment that usually attempts to clarify. I often wonder why people always ask their questions here when they could go right to the source. I am almost sure that Friedman can answer it and it makes so much sense to go to the source especially when it is a question about something written in their book. I could go to the source get the answer and then present it here. $\endgroup$ – Michael R. Chernick May 16 '12 at 1:25
  • 1
    $\begingroup$ People are intimidated by the source's status as an authority, assume the source is far too busy to deal with their (in their opinion) minor and unimportant question, are afraid of getting a rude "why are you bothering me with this?" answer... It's much easier to go to the source if you, too, are a source, perhaps for other stuff, in the field. $\endgroup$ – jbowman May 16 '12 at 1:32
  • 1
    $\begingroup$ @jbowman Yes. I understand that. But you will note that I know Tibshirani and Friedman on a personal basis and assured the Op that their fear is unfounded with these authors. I didn't mention Hastie because I don't know him as well as the others., $\endgroup$ – Michael R. Chernick May 16 '12 at 2:01
  • 1
    $\begingroup$ @chl I don't think we can realistically expect to see them join the site. It requires too much time for busy professors with a few exceptions like Frank Harrell and possibly others who use pseudonyms. But I do think they will take the time to respond to specific questions sent directly to them. $\endgroup$ – Michael R. Chernick May 16 '12 at 20:13
2
$\begingroup$

Let consider a penalized linear model.

The $L_0$ penalty is not very used and is often replaced by the $L_1$ norm that is mathematically more flexible.

The $L_1$ regularization has the property to build a sparse model. This means that only few variables will have a non 0 regression coefficient. It is particularly used if you assume that only few variables have a real impact on the output variables. If there are very correlated variables only one of these will be selected with a non 0 coefficient.

The $L_2$ penalty is like if you add a value $\lambda$ on the diagonal of the input matrix. It can be used for example in situations where the number of variables is larger than the number of samples. In order to obtain a square matrix. With the $L_2$ norm penalty all the variables have non zero regression coefficient.

$\endgroup$
1
  • 2
    $\begingroup$ As an additional contribution, specifically regarding the $L_0$ norm, I don't know that I would say it is because it isn't "mathematically flexible"; I think it is primarily because the optimization is prohibitively expensive (there are ways of trying to do it, but I don't think anything works in complete generality). I know of one "big-cheese" figure who works in variable selection who said he would love to use an $L_0$ penalty and that computation is the only reason he doesn't. $\endgroup$ – guy Jul 5 '14 at 20:42
1
$\begingroup$

A few additions to the answer of @Donbeo

1) The L0 norm is not a norm in the true sense. It is the number of non zero entries in a vector. This norm is clearly not a convex norm and is not a norm in the true sense. Hence you might see terms like L0 'norm'. It becomes a combinatorial problem and is hence NP hard.

2) The L1 norm gives a sparse solution (look up the LASSO). There are seminal results by Candes, Donoho etc. who show that if the true solution is really sparse the L1 penalized methods will recover it. If the underlying solution is not sparse you will not get the underlying solution in cases when p>>n. There are nice results which show that the Lasso is consistent.

3) There are methods like the Elastic net by Zhou and Hastie which combine L2 and L1 penalized solutions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.