4
$\begingroup$

I am learning the concepts of Sparse regression and facing initial hurdles in terminology.

sparse regression model explains the definition of what is meant by sparse. When the number of samples $n$ is less than the signal dimension $p$ then we say it is sparse regression model.

  1. For a model, $x_t = a1x_{t-1} + a2x_{t-2} + white gaussian noise$, the parameters $(a1,a2)$ do not vary with time and for $n=t$ samples we get only two parameters. Then, how come the paper says that $A \in R^{n \times p}$? What does this mean? $A$ is a vector of coefficients and not a matrix. Then what does $R^{n \times p}$ mean?

  2. What I have understood is that there are more number of zero coefficients, but if that so then is there an upper bound which will indicate that the signal is sparse?

  3. I am looking for a reference where I can find such a sparse AR and MA model. Can somebody please point out a link or help in creating such a model?

Thank you

$\endgroup$
3
$\begingroup$

I am working on a research project using Sparse Regression, and what I learned and understood so far is that $\mathbf{A}$ is the input matrix, such that $ \mathbf{A} \in \mathbb{R}^{n \times p}$ where $n$ is number of samples, and $p$ is number of features.

you are trying to find a set of optimal projection vectors $\vec{x_i} \in \mathbb{R}^{p}$ that is mostly zeros, with few nonzero entries, such that the ratio number of nonzero entries and $p$ is your sparseness parameter (typically $s$) such vectors when multiplied by your input matrix $\mathbf{A}$ will discard most of your input features, and will result in a projection $\vec{p_i} = \mathbf{A} \vec{x_i}$ such that, $\vec{p_i} \in \mathbb{R}^{n}$

what I am doing (not sure if this is the standard), is that I find another vector $\vec{\beta}$ to fit a regression model using the projection vectors $\vec{p_i}$ found earlier so that $\hat{y}=\beta_0+\sum_i{\beta_i p_i}$

I hope this helps

$\endgroup$
  • $\begingroup$ Hi, thank you for you reply. By your answer it seems that sparse regression is not different from estimation of signals and parameters except in the data where in sparse there are more zero entries. So, I think you mean to say that research concerns how to get good estimators that perform well by dealing with the sparseness of the data? $\endgroup$ – Ria George Jun 26 '15 at 18:57
  • $\begingroup$ more or less like that. It is somehow used for feature dimensionality reduction (similar to PCA & ICA) $\endgroup$ – A.Rashad Jun 26 '15 at 19:36
2
$\begingroup$

I have been searching for the answer to this question myself and keeping ending up on this thread. I wanted to address your question #2 in case you're still interested or in case others stumble on this post.

I think your understanding of sparse regressions is wrong. I believe that sparse regression is an umbrella term for any regression that penalizes large models and therefore performs variable selection. Examples would be the LASSO, ridge regression, or sparse principal components analysis (which relies on the LASSO).

The "sparse" refers to the fact that the dimension of the parameter vector has been reduced. This is not the same as sparse data! Below is a quote from researchers at Vienna University of Technology:

"The expression 'sparse' should not be mixed up with techniques for sparse data, containing many zero entries. Here, sparsity refers to the estimated parameter vector, which is forced to contain many zeros." http://www.statistik.tuwien.ac.at/public/filz/papers/2011JChem.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.