Multivariate model and large regression

I am not familiar with the concept of multivariate model and just learning about regression model. I am familiar with Autoregressive model and Moving Average.

Multivariate regression model provided cues to look into what a multivariate model is but I could not follow the question as such in this thread. My problem is in the area of sparse regression which I understood as where the coefficient matrix has more number of zero entries.

1. My first question is, the coefficient array should be a vector or why is it a matrix?

2. Is sparse regression only meant for multivariate models? Can a univariate AR(100) be sparse if say 30 of the variables are non-zero lagged. Can somebody show the functional form of how multivariate model looks like and how is it different from sparse if there is any difference?

3. Any references where I can find examples of sparse regression models and books which introduce the concept.

• It's not clear what you mean by "univariate AR(100) be sparse if say 30 of the variables are non-zero lagged". – shadowtalker Sep 10 '14 at 0:59
• Also it sounds like you're really asking about multiple regression (a scalar-valued response with several predictors) and not multivariate regression (a vector-valued response value). Is that accurate? – shadowtalker Sep 10 '14 at 1:06

Consider the multivariate regression model: $$y = \beta_0 + \beta_1 x_1 + \dots + \beta_n x_n$$

Is it a "sparse model"? This question does not make sense, because models cannot be "sparse" by themselves.

Suppose, that you used some data to estimate the parameters of the model. Now if you somehow ended up having $\hat\beta_0 = 1$ and $\hat\beta_i = 0$ for $i\neq 0$, you could say that you got a "sparse solution". It simply means you have many zeroes in the solution vector.

Most importantly, there are certain estimation methods which tend to produce sparse solutions. Namely, if you try to estimate the coefficients for the model above using what is known as "ordinary least-squares (OLS) regression", your solution almost always will not be sparse. Even if the data was generated from a "true" distribution with sparse coefficients, OLS won't recover those, and instead provide you with a bunch of tiny-valued, yet nonzero $\beta_i$.

If, instead of using OLS you will use an $\ell_0$ or $\ell_1$-regularized regression (the latter is famously branded as "LASSO regression"), and you tune the regularization penalty correctly, you may expect to obtain a sparse solution (i.e. many zero-valued $\beta_i$).

To summarize, a "model" cannot be sparse. By "sparsity" people usually mean either a property of a particular estimate, or a property of a whole estimation method. It does not matter what model you are talking about - autoregressive, moving average, linear, nonlinear, etc, the method you use to estimate parameters will define whether you will most probably get a sparse solution or not.

As far as your questions go:

1. It does not really matter how you write down your coefficients. Most often people put them in a vector, but sometimes it may make sense to write them down as a matrix. It depends on the actual model you are working with.

2. The word "sparse" means that your model has many parameters, but the solution sets most of them to zero. Hence yes, it makes no sense to speak about sparsity in the context of models with a single parameter. Often people would care most about sparsity when the number of parameters is in the hundreds of thousands and higher.

3. A kind-of classical paper on the subject is this one. Also, perhaps googling/wikipediing/quoring for terms like "LASSO", "L1-regularization" and "Sparse regression" might provide simpler explanations.