Coordinates of the lasso estimator The lasso estimator is
$$
\hat\beta = \underset{\beta}{\text{argmin}}||Y-X\beta||_2^2+r||\beta||_1
$$
I always read that the coordinates $\hat\beta_j$ of the lasso estimator tend to be either clearly separated from zero or zero exactly. But when I check the outputs of glmnet or other algorithms, I regularly find values that are very small but not exactly zero:  $\hat\beta_j=0.0000018$ for example. Can anybody explain this?
 A: First of all, you should write
$$
\widehat{\beta}\in\text{argmin}_{\beta}||Y-X\beta||_2^2+r||\beta||_1,
$$
because the lasso estimator is not necessarily unique.
Second, you can get intuition about why the coordinates of the lasso estimator tend to be exactly equal to zero by considering orthogonal design ($X^T X=\sqrt{n}I$, where $n$ is the number of samples and $I$ the identity matrix):
the lasso estimator is then the soft-thresholding operator:
$$
\hat{\beta}_j=\text{sign}[X^TY](|X^TY|-r/2)_+/n,
$$
where $(|X^TY|-r/2)_+:=|X^TY|-r/2$ if $|X^TY|>r/2$ and $(|X^TY|-r/2)_+=:0$ otherwise,
see Exercise 2.10 in "Fundamentals of High-Dimensional Statistics: With Exercises and R Labs" for more details.
Hence, the coordinates of the lasso for orthogonal design are exactly zero if and only if $|X^TY|\leq r/2$.
Finally, the reason why algorithms still yield very small values on a regular basis is not that $|X^TY|$ is almost equal to $r/2$ (this can happen but only with very small probability) but that the algorithms only approximate the lasso solution (by coordinate descent in the case of glmnet). In other words, what you see is a numerical phenomenon.
