What is sparse regression model

Question

I am learning the concepts of Sparse regression and facing initial hurdles in terminology.

sparse regression model explains the definition of what is meant by sparse. When the number of samples $n$ is less than the signal dimension $p$ then we say it is sparse regression model.

1. For a model, $x_t = a1x_{t-1} + a2x_{t-2} + white gaussian noise$, the parameters $(a1,a2)$ do not vary with time and for $n=t$ samples we get only two parameters. Then, how come the paper says that $A \in R^{n \times p}$? What does this mean? $A$ is a vector of coefficients and not a matrix. Then what does $R^{n \times p}$ mean?

2. What I have understood is that there are more number of zero coefficients, but if that so then is there an upper bound which will indicate that the signal is sparse?

3. I am looking for a reference where I can find such a sparse AR and MA model. Can somebody please point out a link or help in creating such a model?

Thank you

Answer 1

I am working on a research project using Sparse Regression, and what I learned and understood so far is that $\mathbf{A}$ is the input matrix, such that $ \mathbf{A} \in \mathbb{R}^{n \times p}$ where $n$ is number of samples, and $p$ is number of features.

you are trying to find a set of optimal projection vectors $\vec{x_i} \in \mathbb{R}^{p}$ that is mostly zeros, with few nonzero entries, such that the ratio number of nonzero entries and $p$ is your sparseness parameter (typically $s$) such vectors when multiplied by your input matrix $\mathbf{A}$ will discard most of your input features, and will result in a projection $\vec{p_i} = \mathbf{A} \vec{x_i}$ such that, $\vec{p_i} \in \mathbb{R}^{n}$

what I am doing (not sure if this is the standard), is that I find another vector $\vec{\beta}$ to fit a regression model using the projection vectors $\vec{p_i}$ found earlier so that $\hat{y}=\beta_0+\sum_i{\beta_i p_i}$

I hope this helps

Answer 2

I have been searching for the answer to this question myself and keeping ending up on this thread. I wanted to address your question #2 in case you're still interested or in case others stumble on this post.

I think your understanding of sparse regressions is wrong. I believe that sparse regression is an umbrella term for any regression that penalizes large models and therefore performs variable selection. Examples would be the LASSO, ridge regression, or sparse principal components analysis (which relies on the LASSO).

The "sparse" refers to the fact that the dimension of the parameter vector has been reduced. This is not the same as sparse data! Below is a quote from researchers at Vienna University of Technology:

"The expression 'sparse' should not be mixed up with techniques for sparse data, containing many zero entries. Here, sparsity refers to the estimated parameter vector, which is forced to contain many zeros." http://www.statistik.tuwien.ac.at/public/filz/papers/2011JChem.pdf