Bias of Tibshirani's Lasso estimator I am searching for a theorem that gives upper bounds for the bias of the Lasso estimator from Tibshirani[1]. Do anybody know such a theorem?
[1] Tibshirani, R., (1996).
“Regression Shrinkage and Selection via the Lasso”,
Journal of the Royal Statistical Society. Series B (methodological) 58:(1), p 267–88.
 A: Suppose that we have a rank deficient least squares problem 
$\min \| X\beta -y \|_{2}$
where $\alpha$ is a nonzero vector in the null space of $X$.  That is, 
$X\alpha=0$.  
The lasso estimator can be formulated either as a constrained least squares problem or as unconstrained problem with an objective that includes the least squares term and the one-norm regularization.  I'll used the constrained formulation:
$\min \| X \beta - y \|_{2}^{2} $
subject to 
$\| \beta \|_{1} \leq t $
where $t$ is a fixed regularization parameter.  Let $\beta_{L}$ be the Lasso estimator obtained by solving this constrained optimization problem.  Let $\beta_{2}$ be the minimum 2-norm solution to the least squares problem and suppose that $\| \beta_{2} \|_{1} \leq t$.    
Now consider what happens if the true $\beta$ that we're trying to estimate is of the form 
$\beta=\beta_{2}+ s \alpha$, 
where $s$ is a very large scalar. Since the $s \alpha$ term has no effect on the least squares objective, but greatly increases $\| \beta \|_{1}$, we could have a true $\beta$ that is arbitrarily far from the Lasso estimator and has just as good a least squares objective value as $\beta_{2}$.  
If you use a Bayesian approach you can avoid this issue by specifying a prior that effectively rules out such solutions.    
