How to calculate the information loss of PCA? How would we calculate the information loss of reducing dimensions using PCA ? Would it be the amount of variance loss if we skip certain eigenvectors after the PCA ?
 A: Suppose that we have $\vec y = W \vec x$, where $\vec x \in \mathbb R^n, \vec y \in \mathbb R^m$ and $W$ is some matrix (that could, e.g., by the top $k$ components from PCA). One reasonable way to define the information that $Y$ has about $X$ would be the mutual information, $I(Y;X) = H(X) - H(X | Y)$. The second term is analogous to what you are thinking of as "variance loss". It generalizes the idea that if $Y$ explains variance in X, the mutual information goes up.
If $X$ is Gaussian, i.e., $\vec x \sim \mathcal N(\vec \mu, \Sigma_X)$, then $Y$ is also Gaussian, as is the joint distribution. For a Gaussian distribution, we can exactly write down the formulae for different entropies and we get an expression like this, $I(X;Y) = 1/2 \log \frac{|\Sigma_{X}| |\Sigma_{Y}|}{|\Sigma_{XY}|}$. Whether this lines up exactly with variance loss would depend on how you define it. In the Gaussian case, the mutual information only depends on these covariance matrices, so your intuition is correct and mutual information is directly related to explained variance. 
An alternate perspective on the "information loss" comes from considering the multivariate mutual information, or total correlation, as is explored here.
