Continuous process vs Discrete process! I encountered two statements in Tibshirani's article Regression Shrinkage and Selection via the Lasso, p.58 (pdf).  



*

*Subset selection is a "discrete process" which leads to unstable coefficients.  

*ridge regression is a "continuous process" leading to stable coefficients.


My question is, why ridge is called continuous process while subset selection is a discrete process. Also, why ridge leads to stability and the Subset selection to instability. 
 A: Both subset selection and ridge regression are methods to estimate the coefficients of a linear model.
For subset selection, in its most general form, you pick some metric to optimize and try all possible subsets of variables to include in your linear regression model. Suppose you have $p$ covariates, then there are $2^p$ models, i.e. there are $2^p$ vectors $\beta$ to choose from. Hence, jumping from model to model is a discrete process. You cannot "interpolate" between models.
For ridge regression, you add an $l_2$ penalty to the coefficients, so that the model you choose is
$$ \min_\beta \| X\beta - y\|^2 + \frac{\lambda}{2}\|\beta\|^2$$
In this case, for any $\lambda \ge 0$ you get a different model (that is solution of $\beta(\lambda)$). In fact, $\beta$ will be a smooth function of $\lambda$: change $\lambda$ a little bit, and your fitted $\beta$ will change a little bit. Hence, there are an infinite number of possible models. Thus, this is a continuous process and you can smoothly go from one model to the other by changing $\lambda$.
This is also the reason why subset selection is unstable, and ridge regression is stable. Suppose you change the metric you use to fit the subset selection process a little bit, and you can end up with a very different model (here with different model I mean what elements of $\beta$ are non-zero). As I noted above, this is not true for ridge regression; a small change in $\lambda$ leads to a small change in $\beta$.
Also worth noting that in general all coefficients in Ridge regression are non-zero for all values of $\lambda$, but they converge monotonically to $0$ as $\lambda \to \infty$.
