What is a sparse estimator?

Question

Can anyone point me to a reference, either book or paper, where I can find the precise definition of sparse estimator?

Thanks

Answer 1

Sparse estimators are frequently used in a high dimensional context, namely $p>>n$. Essentially, they offer a regularized version of an estimator e.g. a least squares estimator with an $l_1$ or $l_0$ norm based parameter penalty.

Informally, it promotes zeros in the solution set of the estimator. This is best understood as follows:
Assume you estimate the projected version of $y$ in a subspace of $p$ dimension spanned by the columns of $\mathbf{X}$ given by $\hat{y}= \mathbf{X}\beta$. This equation defines a hyperplane as $\{\beta \in R^p |\mathbf{X}\beta=\hat{y}\}$. Imposing an $l1$ penalty on the allowed values of $\beta$ we define 1-norm ball see figure .
The distance in $l1$ is achieved when the boundary of the ball meets the hyperplane, the solution set will have a minimal cardinality compared to the spherical nature of the $l2$ norm.

In a two dimensional plane this can occur to y-axis which implies a solution of the form $\hat{\beta}=\{0,\hat{\beta_y}\}$, a kind of variable selection.

I think the most extensive treatise of the sparse estimation methods is on the book "Statistics for High Dimensional Data" by Peter Bühlmann.