2
$\begingroup$

Suppose that $X_i, i=1,\ldots,n$ are some random variables. I'd like to do multiple linear regression to learn to predict any of these variables from the others. My model for the reconstructed variables is the following. $$X_i = \sum_{j=1}^n w_{i,j} X_i + \epsilon_i, \forall i=1,\ldots,n$$ A simple thing to do would be to tune the weights, $w$, to minimize the squared error between the reconstruction and the original. Obviously, there is a perfect but trivial solution! We have to enforce that $w_{i,i}=0$ so that we can't predict $X_i$ from itself. After imposing this constraint, we have what I'd call a self-regression problem. Has this simple model been studied and can anyone point me in the right direction?

A related problem would be to use dimensionality reduction to carry out this reconstruction. E.g., we could use an autoencoder with linear encoder and decoder, or simply use PCA. One problem with this point of view is that it is not completely straightforward to predict variable $X_i$ from the other variables $X_{j\neq i}$. Why not? Because the PCA vectors depend on $X_i$ in the first place. Again, I believe this context must be well-studied. We are just doing linear dimensionality reduction with missing variables and we want to know how well we can reconstruct the missing variables. Has this situation been studied?

Improve this question
$\endgroup$
  • $\begingroup$ Please explain what the bar over the $X_i$ means. Obviously it does not have the usual meaning of an arithmetic mean, for then the weights would be known. Regardless of what $\bar X_i$ might be, I don't see where there is anything unusual here: aren't you just posing $n$ separate regression problems? $\endgroup$ – whuber♦ Jun 30 '16 at 2:18
  • $\begingroup$ Good point; originally I meant for the bar to indicate the reconstructed variable, but there's really no need for that. $\endgroup$ – Greg Ver Steeg Jun 30 '16 at 3:22
  • $\begingroup$ Thank you for the clarification. But about my second point: in what way, if any, is this different from a set of $n$ separate regression problems? $\endgroup$ – whuber♦ Jun 30 '16 at 12:39
  • 2
    $\begingroup$ I don't see any specific question, though: of course you know how to do multiple regression. You should edit your post to pose the problem you actually face, because it appears that your formulation has lost the essential character of that problem. Your true interests are emerging only in this last comment. $\endgroup$ – whuber♦ Jun 30 '16 at 14:25
  • 1
    $\begingroup$ I might be wrong, but isn't this just the ansatz of Kaiser's image-analysis? $\endgroup$ – Gottfried Helms Jul 10 '16 at 4:44
1
$\begingroup$

Can we cast this problem as a sparse precision matrix learning problem? It assumes a Gaussian graphical model and learns its structure. See for example,

Ravikumar, P., Wainwright, M. J., Raskutti, G., & Yu, B. (2011). High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence. Electronic Journal of Statistics, 5, 935-980.

There are non-linear extensions of this idea using M-estimators by M. Wainwright. For example, see:

Loh, P. L., & Wainwright, M. J. (2013). Structure estimation for discrete graphical models: Generalized covariance matrices and their inverses. The Annals of Statistics, 41(6), 3022-3049.

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks Taha, I think you're right: the easiest way to look at this is essentially as an inverse covariance matrix estimation problem (that shows up in the M matrix in my answer). I'm speculating that the reason that this self-regression problem doesn't show up a lot is because this inverse covariance problem is so hard. The only way to learn decent linear imputations is probably through imposing sparsity, as you suggest. $\endgroup$ – Greg Ver Steeg Jul 2 '16 at 23:03
  • $\begingroup$ Also relevant in that context is this paper: arxiv.org/pdf/0903.5463v3.pdf which shows how to do this with missing values. $\endgroup$ – Greg Ver Steeg Jul 2 '16 at 23:07
  • $\begingroup$ We can show that the non-zeros in the precision matrix correspond to the edges in the graphical model which makes it more attractive that covariance matrix learning. I missed your answer below before writing this answer, which is why they are essentially the same. P. Ravikumar has worked a lot on the sparse inverse covariance matrix estimation; especially on scaling it up. Thank you for the reference. $\endgroup$ – Taha Jul 3 '16 at 1:58
0
$\begingroup$

Update: Inverse covariance and regression The connection turns out to be very straightforward. The self regression coefficients can be directly extracted from the inverse covariance matrix. I finally stumbled on literature addressing this connection starting in this paper.

March 2017 update: this paper is even better at explaining the connection.

Old answers below

I finally found a reference to the exact formula written above. It appears in this paper in Eq. 3.5. Hyvarinen also refers to this as a special case of structural equation modeling (SEM). However, he points out that this special case is the basis of the famous causal discovery framework LINGAM (with a few extra restrictions required). Hyvarinen lists a few other related works there.

Here are three related things I mentioned before I found the reference above.

Covariance estimation Let each sample, $\vec x$, be drawn iid from a multivariate normal distribution, $\vec x \sim \mathcal N(\vec 0, \Sigma)$. In that case, the distribution of each $X_i$ conditioned on the other variables, $X_{j\neq i}$, is just a normal distribution with mean, $\mu_i$, and variance, $\sigma_i^2$, that can be constructed from $\Sigma$ using standard identities. Use $\mathbb E$ to denote expectation values (and keep in mind that for simplicity and w.l.o.g. we considered random variables with zero mean). We will define $X_{j \neq i} \equiv X_1, \ldots, X_{i-1},X_{i+1},\ldots,X_n$ for compact notation. $$\mathbb E [X_i |X_{j\neq i} ] = \sum_{j, k \neq i}^n \Sigma_{i,j}~ M_{j,k}^i ~X_k $$ $$\mathbb E [X_i^2 |X_{j\neq i} ] = \mathbb E [ X_i^2 ] - \sum_{j, k \neq i}^n \Sigma_{i,j} ~ M_{j,k}^i ~\Sigma_{k,i} $$ The matrix $M^i$ is formed by first deleting row and column $i$ of $\Sigma$, and then taking the matrix inverse of the result. The first line tells us how to predict $X_i$ from the other values and the second how uncertain that prediction is.

Dual total correlation Considering instead a more general point of view, the uncertainty in a variable, $X_i$, conditioned on the other observations, can be written as the conditional entropy, $H(X_i |X_{j\neq i})$. The sum of these conditional entropies appears in the dual total correlation, which obeys various bounds, in particular with respect to the total correlation.

Linear imputation One domain where this has been explicitly studied is for imputing missing values based on a linear model. This is implemented in Stata, where I see this book cited. I haven't had a chance to check this out yet.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.