I'm using covariance matrices to do a linear regression. I have the same number of predictors and dependent variables (3000 of each). For each $i$, $X_{i}$ and $Y_{i}$ have a kind of symmetry.

I gather covariance matrices of these variables ($X'X$ and $X'Y$). I want to do a regression where for each dependent variable, the features are the first 1,000 dependent variables plus its own dependent variable.

For example, to predict $Y_{2000}$ I want to use $X_{1},...,X_{1000}$ plus $X_{2000}$, etc. I could do this by inverting 3000 different matrices. But that is computationally expensive. Is there a way to do this more efficiently by overlapping the computation?

  • $\begingroup$ You’re using $Y_i$ as a predictor of $Y_i$? $\endgroup$
    – Dave
    Commented Apr 18, 2020 at 16:18
  • $\begingroup$ I'm using X.1 .. X1000 PLUS Xi as a predictor of Yi. I actually just figured out the answer to my question. It's here en.wikipedia.org/wiki/Sherman%E2%80%93Morrison_formula $\endgroup$ Commented Apr 18, 2020 at 23:20

1 Answer 1


With multiple predictors and multiple dependent variables, you might consider partial least-squares regression, also known as projection to latent structures (PLS, in either case).

As the Wikipedia page linked above puts it,

A PLS model will try to find the multidimensional direction in the X [predictor] space that explains the maximum multidimensional variance direction in the Y [dependent variable] space.

PLS thus takes into account correlations both among predictors and among the dependent variables while modeling the joint relationships among the predictors and the dependent variables. That approach would avoid separate fits for all 3000 dependent variables while still allowing you to re-project predictions back to the original set of dependent variables once an adequate model is obtained.

Some implementations of PLS are restricted to a situation with only a single dependent variable, so you will need to use an implementation that allows for multiple dependent variables. The Wikipedia page has a link to an implementation based on singular-value decompositions, which could improve efficiency with your high-dimensional sets of predictors and dependent variables.

Some of the over 200 questions with the partial-least-squares tag on this site might provide further help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.