Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am searching for a theorem that gives upper bounds for the bias of the Lasso estimator from Tibshirani[1]. Do anybody know such a theorem?
[1] Tibshirani, R., (1996).
“Regression Shrinkage and Selection via the Lasso”,
Journal of the Royal Statistical Society. Series B (methodological) 58:(1), p 267–88.
Suppose that we have a rank deficient least squares problem
$\min \| X\beta -y \|_{2}$
where $\alpha$ is a nonzero vector in the null space of $X$. That is, $X\alpha=0$.
The lasso estimator can be formulated either as a constrained least squares problem or as unconstrained problem with an objective that includes the least squares term and the one-norm regularization. I'll used the constrained formulation:
$\min \| X \beta - y \|_{2}^{2} $
subject to
$\| \beta \|_{1} \leq t $
where $t$ is a fixed regularization parameter. Let $\beta_{L}$ be the Lasso estimator obtained by solving this constrained optimization problem. Let $\beta_{2}$ be the minimum 2-norm solution to the least squares problem and suppose that $\| \beta_{2} \|_{1} \leq t$.
Now consider what happens if the true $\beta$ that we're trying to estimate is of the form
$\beta=\beta_{2}+ s \alpha$,
where $s$ is a very large scalar. Since the $s \alpha$ term has no effect on the least squares objective, but greatly increases $\| \beta \|_{1}$, we could have a true $\beta$ that is arbitrarily far from the Lasso estimator and has just as good a least squares objective value as $\beta_{2}$.
If you use a Bayesian approach you can avoid this issue by specifying a prior that effectively rules out such solutions.
