I'm looking for some sort of guideline as when it is appropriate to use which forms of regularization and a comparison of the advantages / disadvantages of the various forms. So something that compares the likes of ridge regression with LASSO or elastic net etc and forward/backward/stepwise selection using AIC/BIC. What about other methods that can achieve similar goals like Principal Component Regression?
Are there any review papers on the topic?
From The Elements of Statistical Learning, as suggested by goangit, section 3.6 is a one page discussion comparing selection and shrinkage methods which points to a paper by Frank and Friedman (1993) A Statistical View of Some Chemometrics Regression Tools. Section 5 of their paper (page 125) performed a Monte Carlo study comparing Ordinary Least Squares (OLS), Ridge Regression (RR), Principal Component Regression (PCR), Partial Least Squares (PLS) and Variable Subset Selection (VSS). They conclude that in terms of prediction accuracy, although all methods do outperform OLS, RR is superior under a variety of conditions (all those tested). VSS offers the lowest increase in accuracy with PCR and PLS not far behind RR. Section 8 is a brief discussion over the descriptive properties of the method. Although no formal conclusions are reachable since this is a subjective matter, they do say that VSS and PLS may have advantages if this is a goal of the study. Unfortunately, they do not include LASSO in their study.
You could try Introduction to Statistical Learning by Gareth James et al. It's freely available, contains introductory-level review and discusion of all the topics you mention (in particular Chapter 6 deals with regularization), is well-supported by the ISLwR package in R and provides a gateway to the more advanced counterpart The Elements of Statistical Learning by Hastie et al.
