This is a specific application of the usual least squares derivation for linear regression but I'm confusing myself.
Suppose $X = (X_1, ..., X_d)$ is a random vector.
Pick any $i, j < d$ and let $X_{-ij}$ be $X$ without the $i$ and $j$ components.
Then we want to regress on $X_{-ij}$ to find $X_i$ and $X_j$.
So I want to find $\beta_i$ and $\beta_i^0$ such that $X_{-ij}^T\beta_i + \beta_i^0$ minimizes the expected least square error.
i.e, I want to find $\text{argmin}_{\beta_i, \beta_i^0}\mathbb{E}[(X_i - X^T_{-ij}\beta_i - \beta_i^0)^2]$ and similar for $\beta_j, \beta_j^0$
$\beta_i, \beta_i^0$ will be in terms of variances and covariances of the $X_k$ once I'm done.
I've been unable to get the terms to behave however, and have unable to find a solid expression for the $\beta_i$. Would appreciate help with the derivation.
