There are many methods to perform regularization -- $L_0$, $L_1$, and $L_2$ norm based regularization for example. According to Friedman Hastie & Tibsharani, the best regularizer depends on the problem: namely the nature of the true target function, the particular basis used, signal-to-noise ratio, and sample size.
Is there any empirical research comparing methods and performance of various regularization methods?
Let consider a penalized linear model.
The $L_0$ penalty is not very used and is often replaced by the $L_1$ norm that is mathematically more flexible.
The $L_1$ regularization has the property to build a sparse model. This means that only few variables will have a non 0 regression coefficient. It is particularly used if you assume that only few variables have a real impact on the output variables. If there are very correlated variables only one of these will be selected with a non 0 coefficient.
The $L_2$ penalty is like if you add a value $\lambda$ on the diagonal of the input matrix. It can be used for example in situations where the number of variables is larger than the number of samples. In order to obtain a square matrix. With the $L_2$ norm penalty all the variables have non zero regression coefficient.
A few additions to the answer of @Donbeo
1) The L0 norm is not a norm in the true sense. It is the number of non zero entries in a vector. This norm is clearly not a convex norm and is not a norm in the true sense. Hence you might see terms like L0 'norm'. It becomes a combinatorial problem and is hence NP hard.
2) The L1 norm gives a sparse solution (look up the LASSO). There are seminal results by Candes, Donoho etc. who show that if the true solution is really sparse the L1 penalized methods will recover it. If the underlying solution is not sparse you will not get the underlying solution in cases when p>>n. There are nice results which show that the Lasso is consistent.
3) There are methods like the Elastic net by Zhou and Hastie which combine L2 and L1 penalized solutions.
