Which has more mutual information with a multivariate Gaussian: its first principal component, or its first factor? I have a $k$-dimensional Gaussian random variable $X\sim\mathcal{N}(0, \Sigma_X)$. What I want is a 1-dimensional scalar r.v. $Y\sim\mathcal{N}(0,1)$ that is jointly Gaussian with $X$ while maximizing $I(X; Y)$ (the mutual information between $X$ and $Y$).
Another way to phrase this is I want to find a set of weights $c\in\mathbb{R}^k$ such that $Y = c^T X$ maximizes $I(X ; Y)$. 
One candidate for $c$ is the vector that projects onto the 1st principal component of $\Sigma_X$ (namely, the [scaled] eigenvector corresponding to the largest eigenvalue of $\Sigma_X$).
Another candidate is to use factor analysis to find a scalar factor, which is like choosing a $c$ such that $\Sigma_X - c c^T$ is as close to diagonal as possible.
Which of these (if either) would maximize the mutual information?
My intuition is that the 1D factor would be the answer, but the following analysis has led me to some weird conclusions:
From Wikipedia, the mutual information between two Gaussians $X$ and $Y$ is $$-\frac{1}{2}\ln(|\rho|)$$, where $\rho$ is the joint correlation matrix between $X$ and $Y$. Using Schur's determinant identity for block matrices, it's not hard to see that, in our case, $$|\rho| = \left|\rho_X - \gamma \gamma^T\right|$$, where $\rho_X$ is the correlation matrix for $X$, and $c = \gamma \otimes \sqrt{diag(\Sigma_X)}$ (here, $\otimes$ is element-wise multiplication). 
What this implies is that, if we choose $\gamma$ to be an appropriately scaled version of any of the eigen-vectors of $\rho_X$, then the determinant (product of eigenvalues) will be 0, implying infinite (perfect) mutual information between $Y$ and $X$. This...doesn't seem to be right, since wouldn't that mean that even the weakest principle component of $X$ would not only maximize, but give perfect mutual information?
 A: I realize this question is three years old, but I just came across this during my own Googling, so here goes!
The simple answer is that, mutual information is not really the metric you want here, because of the oddities of differential entropy. Mutual information is best thought of as measurements of an unknown system, and trying to learn the most possible: the amount of bits you read out is the mutual information. In this case, when you take any 1D projection of your random vector, you're getting an exact real number worth of information about it. This has, really, infinitely many bits, hence why the mutual information is infinite.
Put differently, in terms of the determinant: if I have n-dimensional multivariate Gaussian and get some noisy measurement, then I update my belief distribution and shrink the Gaussian to some degree, narrowing it. The entropy tells me about the volume of that ellipsoidal region, which I most suspect the true value to be in, the  1-sigma region. I have a volume I believe the point belongs to, the logarithm of that volume is the differential entropy.
If I squeeze that ellipsoid down to zero width along any one axis, the volume becomes zero, and the differential entropy approaches $-\infty$. (Remember, differential entropy can be negative.) This is what happens when you get your one perfect scalar measurement. If your measurement had a very small amount of noise $\epsilon$, then your mutual information would be approximately equal to some constant $ - \log(\epsilon)$, indicating that extra bit of uncertainty.
So what do you do in this situation, where you want to pick the most useful measurement? Well, if "zero volume" doesn't actually reflect "no uncertainty" for your application, then differential entropy is not the figure of merit you should care about. You might instead care about the variance of the point, $\sum_k E[(X_k - E[X_k])^2]$, or maybe there's some other natural error bound you actually want to use.
If you really want to minimize differential entropy, then any perfect measurement will get you $-\infty$, and you could be happy with any one of them. If your perfect measurement has some tiny error $\epsilon$ (floating point error? quantum uncertainty? ), then the optimal measurement choice would be along the principal component: the resulting projected Gaussian would be as small as possible, across any linear measurement.
