As far as I know, LARS solves the following problem (using the same notation as Efron et al. Least angle regression):

Given a vector y, and a matrix X. Pick some column vectors from X, and express y as a linear combination of these column vectors.

In my problem, I have a set of vectors, which can form a matrix Y. I would like to express each column vector of Y as a linear combination of some column vectors from X. I hope that the fitting quality could be good (L2), and the number of column vectors I use from X is small (L1). Therefore, I am wondering if there is any systematic study (any papers?) on applying LARS to fit a set of vectors, rather than just one vector.

Below is my solution. Comments are highly appreciated.

Suppose Y = (y1 y2 ...) and X = (x1 x2 ...), where yi, xi are vectors. Treat Y as a high-dimensional 1D vector (flatten it) like

$Y' = \left(\begin{array}{c} y1\\ y2\\ ...\end{array}\right)$

Then we define the new basis matrix X' as (the blank part are all zeros)

$X' = \left(\begin{array}{ccc} X & & \\ & X &\\ & & ...\end{array}\right)$

Then we just apply LARS directly to fit Y' using X'. Once we have picked columns from X', we merge them into a set of columns from X, which will be the final "basis vector" set to fit Y.

One thing I feel uncomfortable about the above solution is that, it does not consider the fact that the same set of basis vectors from X will be used to express all column vectors of Y. Only the final step merges the result from LARS. Your help is appreciated. Thanks.

  • $\begingroup$ The implication seems to be that you want to select the same columns of $X$ for all the response vectors, but allow for a different linear combination of the former for each of the latter. Is this your intent? $\endgroup$ – cardinal Feb 22 '12 at 16:04

You should read about multi-task learning. A good place to start is this page: http://www.public.asu.edu/~jye02/Software/MALSAR/index.html (both the manual and tutorial).

There are many, many formulations and regularizers that people have devised for the class of problems you are interested in.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.