# Least Squares with respect to a Matrix

There is a nice geometric interpretation of ordinary least squares where $\beta \in\mathbb{R}^p$ is to be learned/ estimated from data given by $y \in\mathbb{R}^n$ and $x \in\mathbb{R}^{n\times p}$ with $n\geq p$ known, so that

$$\hat{\beta} = \underset{\beta}{\text{argmin}}\ ||y-x \beta||_2^2 = (x^Tx)^{-1}x^Ty.$$

I can setup a similar looking problem where now I say $\beta_i$ and $y_i$ are known vectors for $1\leq i \leq N$ and I wish to estimate the design matrix $x$ as

$$\hat{x} = \underset{x}{\text{argmin}}\ \sum_{i=1}^N||y_i-x\beta_i||_2^2 = \left(\sum_{i=1}^Ny_i\beta_i^T\right)\left(\sum_{i=1}^N\beta_i\beta_i^T\right)^{-1}\,,$$

where $N\geq p$ is required for the inverse to exist. (This was solved using the critical point that solves the score function.) I'm not sure how to interpret this result geometrically anymore. Could someone please explain this to me?

• I have taken the liberty of editing this question to put it in standard notation for OLS in regression, which is the notation most familiar to statistical readers on this site. There was no error in your original question, but the notation was harder to understand, since some of the standard notation for particular objects was used for other objects. If you prefer your original notation, please feel free to reverse my edits.
– Ben
Commented May 18, 2018 at 0:18

## 1 Answer

For a matrix $X \in \mathbb R^{n \times p}$ note that $$X^TX = \sum_{i=1}^n x_ix_i^T$$ where $x_i$ is the $i$th row of $X$.

Collect your $\beta_i$ into the rows of a matrix $B \in \mathbb R^{N \times p}$. Then we can write $$B^TB = \sum_{i=1}^N \beta_i\beta_i^T.$$

Now let $y_i \in \mathbb R^n$ be the response vector for $i = 1, \dots, N$ and collect these into the rows of the matrix $Y \in \mathbb R^{N\times n}$. Then $$\sum_i y_i \beta_i^T = Y^TB$$ so all together $$\hat x = Y^TB(B^TB)^{-1}$$ which is just the transpose of the usual multivariate regression of $Y$ on $B$ which we know how to interpret (although I can add more details if you'd like).