Least Squares with respect to a Matrix

Question

There is a nice geometric interpretation of ordinary least squares where $\beta \in\mathbb{R}^p$ is to be learned/ estimated from data given by $y \in\mathbb{R}^n$ and $x \in\mathbb{R}^{n\times p}$ with $n\geq p$ known, so that

$$\hat{\beta} = \underset{\beta}{\text{argmin}}\ ||y-x \beta||_2^2 = (x^Tx)^{-1}x^Ty.$$

I can setup a similar looking problem where now I say $\beta_i$ and $y_i$ are known vectors for $1\leq i \leq N$ and I wish to estimate the design matrix $x$ as

$$\hat{x} = \underset{x}{\text{argmin}}\ \sum_{i=1}^N||y_i-x\beta_i||_2^2 = \left(\sum_{i=1}^Ny_i\beta_i^T\right)\left(\sum_{i=1}^N\beta_i\beta_i^T\right)^{-1}\,,$$

where $N\geq p$ is required for the inverse to exist. (This was solved using the critical point that solves the score function.) I'm not sure how to interpret this result geometrically anymore. Could someone please explain this to me?

Answer 1

For a matrix $X \in \mathbb R^{n \times p}$ note that $$ X^TX = \sum_{i=1}^n x_ix_i^T $$ where $x_i$ is the $i$th row of $X$.

Collect your $\beta_i$ into the rows of a matrix $B \in \mathbb R^{N \times p}$. Then we can write $$ B^TB = \sum_{i=1}^N \beta_i\beta_i^T. $$

Now let $y_i \in \mathbb R^n$ be the response vector for $i = 1, \dots, N$ and collect these into the rows of the matrix $Y \in \mathbb R^{N\times n}$. Then $$ \sum_i y_i \beta_i^T = Y^TB $$ so all together $$ \hat x = Y^TB(B^TB)^{-1} $$ which is just the transpose of the usual multivariate regression of $Y$ on $B$ which we know how to interpret (although I can add more details if you'd like).