Linear "self" regression, terminology and references? Suppose that $X_i, i=1,\ldots,n$ are some random variables. I'd like to do multiple linear regression to learn to predict any of these variables from the others. My model for the reconstructed variables is the following. $$X_i = \sum_{j=1}^n w_{i,j} X_i + \epsilon_i, \forall i=1,\ldots,n$$ 
A simple thing to do would be to tune the weights, $w$, to minimize the squared error between the reconstruction and the original. Obviously, there is a perfect but trivial solution! We have to enforce that $w_{i,i}=0$ so that we can't predict $X_i$ from itself. After imposing this constraint, we have what I'd call a self-regression problem. Has this simple model been studied and can anyone point me in the right direction? 
A related problem would be to use dimensionality reduction to carry out this reconstruction. E.g., we could use an autoencoder with linear encoder and decoder, or simply use PCA. One problem with this point of view is that it is not completely straightforward to predict variable $X_i$ from the other variables $X_{j\neq i}$. Why not? Because the PCA vectors depend on $X_i$ in the first place. Again, I believe this context must be well-studied. We are just doing linear dimensionality reduction with missing variables and we want to know how well we can reconstruct the missing variables. Has this situation been studied?
 A: Update: Inverse covariance and regression 
The connection turns out to be very straightforward. The self regression coefficients can be directly extracted from the inverse covariance matrix. I finally stumbled on literature addressing this connection starting in this paper. 
March 2017 update: this paper is even better at explaining the connection.
Old answers below
I finally found a reference to the exact formula written above. It appears in this paper in Eq. 3.5. Hyvarinen also refers to this as a special case of structural equation modeling (SEM). However, he points out that this special case is the basis of the famous causal discovery framework LINGAM (with a few extra restrictions required). Hyvarinen lists a few other related works there. 
Here are three related things I mentioned before I found the reference above.
Covariance estimation
Let each sample, $\vec x$, be drawn iid from a multivariate normal distribution, $\vec x \sim \mathcal N(\vec 0, \Sigma)$. In that case, the distribution of each $X_i$ conditioned on the other variables, $X_{j\neq i}$, is just a normal distribution with mean, $\mu_i$, and variance, $\sigma_i^2$, that can be constructed from $\Sigma$ using standard identities. Use $\mathbb E$ to denote expectation values (and keep in mind that for simplicity and w.l.o.g. we considered random variables with zero mean). We will define $X_{j \neq i} \equiv X_1, \ldots, X_{i-1},X_{i+1},\ldots,X_n$ for compact notation. 
$$\mathbb E [X_i |X_{j\neq i} ] = \sum_{j, k \neq i}^n \Sigma_{i,j}~ M_{j,k}^i ~X_k $$
$$\mathbb E [X_i^2 |X_{j\neq i} ] =  \mathbb E [ X_i^2 ] - \sum_{j, k \neq i}^n \Sigma_{i,j} ~ M_{j,k}^i ~\Sigma_{k,i} $$
The matrix $M^i$ is formed by first deleting row and column $i$ of $\Sigma$, and then taking the matrix inverse of the result.
The first line tells us how to predict $X_i$ from the other values and the second how uncertain that prediction is. 
Dual total correlation Considering instead a more general point of view, the uncertainty in a variable, $X_i$, conditioned on the other observations, can be written as the conditional entropy, $H(X_i |X_{j\neq i})$. The sum of these conditional entropies appears in the dual total correlation, which obeys various bounds, in particular with respect to the total correlation. 
Linear imputation One domain where this has been explicitly studied is for imputing missing values based on a linear model. This is implemented in Stata, where I see this book cited. I haven't had a chance to check this out yet. 
A: Can we cast this problem as a sparse precision matrix learning problem? It assumes a Gaussian graphical model and learns its structure.  See for example, 
Ravikumar, P., Wainwright, M. J., Raskutti, G., & Yu, B. (2011). High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence. Electronic Journal of Statistics, 5, 935-980.
There are non-linear extensions of this idea using M-estimators by M. Wainwright. For example, see:
Loh, P. L., & Wainwright, M. J. (2013). Structure estimation for discrete graphical models: Generalized covariance matrices and their inverses. The Annals of Statistics, 41(6), 3022-3049. 
