Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Can anyone point me to a reference, either book or paper, where I can find the precise definition of sparse estimator?


share|improve this question
Not sure, but maybe sparsity of an estimator (vector valued) $\hat{\theta}$ means that most off-diagonal entries of the covariance matrix are zeros, so components of $\hat{\theta}$ are mostly orthogonal. Maybe you should look for "orthogonal parametrization" in the literature. – Matteo Fasiolo Nov 22 '12 at 13:49

Sparse estimators are frequently used in a high dimensional context, namely $p>>n$. Essentially, they offer a regularized version of an estimator e.g. a least squares estimator with an $l_1$ or $l_0$ norm based parameter penalty.

Informally, it promotes zeros in the solution set of the estimator. This is best understood as follows:
Assume you estimate the projected version of $y$ in a subspace of $p$ dimension spanned by the columns of $\mathbf{X}$ given by $\hat{y}= \mathbf{X}\beta$. This equation defines a hyperplane as $\{\beta \in R^p |\mathbf{X}\beta=\hat{y}\}$. Imposing an $l1$ penalty on the allowed values of $\beta$ we define 1-norm ball see figure .
The distance in $l1$ is achieved when the boundary of the ball meets the hyperplane, the solution set will have a minimal cardinality compared to the spherical nature of the $l2$ norm.

In a two dimensional plane this can occur to y-axis which implies a solution of the form $\hat{\beta}=\{0,\hat{\beta_y}\}$, a kind of variable selection.

I think the most extensive treatise of the sparse estimation methods is on the book "Statistics for High Dimensional Data" by Peter Bühlmann.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.