LASSO with poorly conditioned predictors I need to solve equations of the form $Ax=b$, with $A$ a $m\times n$ matrix with $m>n$.
I am looking at the usual least squares solution $x_0=A^+b$, where $A^+$ is the pseudoinverse and at the LASSO solution $x_L$.
I have observed that when the smallest singular value of $A$ becomes very small, $x_0$ becomes wild, with very large elements. This is of course no surprise at all. What is surprising is that $x_L$ is still very well behaved.
Why is the LASSO solution almost insensitive to the ill conditioning of $A$? 
EDIT: I have consulted the book "Elements of Statistical Learning", as suggested, but I could not find an explanation in there to my question. I understand why the ridge algorithm performs well when $A$ is ill conditioned: it is because the solution to ridge involves not the inverse of $A$ but the inverse of $A+\lambda$, so this regularizes the solution. I would like to get a similar understanding in the case of LASSO.
 A: When you say you're solving $Ay = b$, you're automatically placing yourself into a normal OLS regression problem. Lasso simply does not belong to that kind of problem, which is why "the matrix being ill conditioned" has little meaning. 
By solving a normal OLS, you're finding $b$ that minimizes $RSS(b)=\sum(y_i - b_0 - \sum_jx_{ij}b_j)^2.$ You rewrite this as: 
$RSS(b) = (y-Xb)^T(y-Xb)$  which has solution 
$b = (X^TX)^{-1}X^Ty = Ay$ 
which is the linear programming problem you mentioned, where the matrix $A$ is ill conditioned.

When you switch to lasso, the problem becomes: 
$RSS(b) = (y-Xb)^T(y-Xb) + \lambda b$

This would end up being: 
$b = (X^TX)(2X^Ty + \lambda)$ 
which simply CANT be written in a $Ay = b$ form, so it's impossible to see how the original matrix $A$ affects the problem. 

The fact that $(X^TX)^{-1}X^T$ is ill conditioned has no impact on the LASSO problem, as it simply solves a different problem which is quadratic! (and matrix ill conditioning is only defined for linear systems)
