What are some of the most important "early papers" on Regularization methods?

Question

In several answers I have seen CrossValidated users suggest OP find early papers on Lasso, Ridge, and Elastic Net.

For posterity, what are the seminal works on Lasso, Ridge, and Elastic Net?

Answer 1

Since you're simply looking for references, here is the list:

1. Tikhonov, Andrey Nikolayevich (1943). "Об устойчивости обратных задач" [On the stability of inverse problems]. Doklady Akademii Nauk SSSR. 39 (5): 195–198.
2. Tikhonov, A. N. (1963). "О решении некорректно поставленных задач и методе регуляризации". Doklady Akademii Nauk SSSR. 151: 501–504.. Translated in "Solution of incorrectly formulated problems and the regularization method". Soviet Mathematics. 4: 1035–1038.
3. Hoerl AE, 1962, Application of ridge analysis to regression problems, Chemical Engineering Progress, 1958, 54–59.
4. Arthur E. Hoerl; Robert W. Kennard (1970). "Ridge regression: Biased estimation for nonorthogonal problems". Technometrics. 12 (1): 55–67. doi:10.2307/1267351. https://pdfs.semanticscholar.org/910e/d31ef5532dcbcf0bd01a980b1f79b9086fca.pdf
5. Tibshirani, Robert (1996). "Regression Shrinkage and Selection via the Lasso" (PostScript). Journal of the Royal Statistical Society, Series B. 58 (1): 267–288. MR 1379242 https://statweb.stanford.edu/~tibs/lasso/lasso.pdf
6. Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B. 67: pp. 301–320. https://web.stanford.edu/~hastie/Papers/B67.2%20%282005%29%20301-320%20Zou%20&%20Hastie.pdf

Answer 2

A historically important paper which I believe first demonstrated that biasing estimators can result in improved estimates for ordinary linear models:

• Stein, C., 1956, January. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the Third Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 399, pp. 197-206).

A few more modern and important penalties include SCAD and MCP:

• Fan, J. and Li, R., 2001. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456), pp.1348-1360.
• Zhang, C.H., 2010. Nearly unbiased variable selection under minimax concave penalty. The Annals of statistics, 38(2), pp.894-942.

And some more on very good algorithms for obtaining estimates using these methods:

• Breheny, P. and Huang, J., 2011. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The annals of applied statistics, 5(1), p.232.
• Mazumder, R., Friedman, J.H. and Hastie, T., 2011. Sparsenet: Coordinate descent with nonconvex penalties. Journal of the American Statistical Association, 106(495), pp.1125-1138.

Also worth looking at is this paper on the Dantzig selector which is very closely related to the LASSO, but (i believe) it introduces the idea of oracle inequalities for statistical estimators which are a pretty powerful idea

• Candes, E. and Tao, T., 2007. The Dantzig selector: Statistical estimation when p is much larger than n. The Annals of Statistics, pp.2313-2351.