# Whitening a dataset with fewer observations than variables

I have a k x n dataset where k equals the number of variables and n equals the number of observations per variable. I know these data are correlated and I would like to whiten them with the ordinary whitening transformation. Inconveniently, k outnumbers n by far, so that when estimating the k x k covariance matrix it will not be invertible. To get around this problem, I estimated the covariance matrix using optimal shrinkage, but the obtained covariance matrix is not suited for whitening anymore.

I'd be grateful for ideas on how to whiten these data.

• In order to get sensible advice here, you will need to explain why you want to "whiten" your data --- what is your broader purpose for this data, by which you can evaluate if a method "works"?
– Ben
Nov 1, 2022 at 1:01

I am doing the working assumption that by "whitening", we mean "sphereing" in the sense that data is uncorrelated and of equal variance along each dimension (not just to remove the mean and rescale the variance to 1 per dimension). I also note that the even when we are in $$n > p$$ setting the whitening matrix itself is non-unique; the three main ones are PCA whitening, ZCA whitening (sometimes called Mahalanobis whitening) and Cholesky whitening. The PCA is likely the most commonly used but the other are equally correct.

While anti-climatic I don't think there is a simple solution to this when $$p > n$$, $$n$$ being the number of observations and $$p$$ being the number of variables/dimensions/coordinates. That is because the information for the rescaling all the variables while accounting for the covariances simply isn't there.

The simplest sphering can be done via the Cholesky decomposition, we effectively take the covariance matrix $$C$$, invert it to get $$C^{-1}$$, decomposite as $$C^{-1}=LL^T$$ and use $$L^T$$ as the "sphering matrix". In the case of $$C$$ being rank-deficient though, the inverse doesn't exist so our immediate avenue is stopped. As suggested we can try using the Moore–Penrose pseudo-inverse to get our pseudo-inverse $$C^{+}$$ such that $$CC^{+}C=C$$. That is unfortunately a dead-end because the Cholesky decomposition still needs a positive definite (PD) matrix (we can actually get some little extra mileage out of the $$LDL^T$$ decomposition as it is willing to play ball with positive semi-definite (PSD) matrices but as soon as $$p - n> 1$$ things break down again). The pseudo-inverse $$C^{+}$$ is not PD so Cholesky whitening is a no-go.

Next we try ZCA; ZCA actually has two equivalent formulations, one based on the inverse of the covariance matrix and one based on SVD ($$USV^T = X$$), the sphering matrices for those are $$C^{-\frac{1}{2}}$$ (that's why it is called Mahalanobis) and $$VS^{-1}V^T$$ respectively. The case of $$C^{-\frac{1}{2}}$$ is a no-go as described above - even if we get the inverse getting the square root of it is impossible as that matrix is not PD. The SVD one is the more promising as we do not invert anything directly but then we are faced with another problem: despite it's best intentions, the diagonal matrix $$S$$ is $$n \times n$$, not $$p \times p$$. That means that simply put, we cannot rescale $$p$$ varibles with $$n$$ variances if $$n < p$$ ($$V$$ will be $$n\times p$$ so that's fine). So that again doesn't work.

Finally we have the PCA variant; it uses the eigendecomposition of the covarinace matrix $$C = U\Lambda U^T$$ and the sphrering matrix is simply $$\Lambda^{-\frac{1}{2}}U^T$$. Now, if $$C$$ is rank-deficient we will still get $$p$$ eigenvalues which is a step forward from the SVD above but those eigen-values are numerically zero as $$C$$ is (at best) PSD. Effectively the eigendecomposition comes back deficient by itself. Granted we can set small/numerically zero eigenvalues to something like $$\epsilon$$ and get the whole pipeline working but the resulting sphrered data will be have $$p-n$$ columns with near $$\epsilon$$ variance and likely near $$\epsilon$$ covariance with the other columns. The reason for that is that by "fixing" the numerically zero eigenvalues we zero-ed the variance along these components/coordinates.

To recap, we cannot directly whiten/make spherical a rank-deficient matrix using any of the standard sphrering methodologies. Some methodologies do not compute at all (e.g. Cholesky whitening) and some truncate the dimensions that have variabillity (e.g. PCA whitening). I used Optimal whitening and decorrelation by Kessy et al. (2018) as my standard reference. Do note that we can still using ridge or LASSO regression if we want to use that data further, we just need to standardise the columns in that case which is by itself much easier to do.

You didn't tell us why you want to whiten the data, but, however, you could simply use the generalized inverse of the sample covariance matrix the same way you uses its inverse (when it exists). It should have the same effect. (I can add details if wanted)

• Unfortunately, this answer doesn't provide enough detail to be of any use. It's not clear what is meant by "use the generalized inverse of the sample covariance matrix the same way you use its inverse." There's also no justification for why the proposed method should work. Oct 30, 2022 at 23:06
• OK, I will try to add details Oct 30, 2022 at 23:41
• Hmm... I think this is not very correct because even if we calculate the generalised inverse of the potentially rank-deficient covariance matrix, we will still have a rank-deficient inverse so doing for example Cholesky on it to use $L^T$ as the sphering matrix will still fail. Sure we can use other sphering like ZCA where we can use the SVD $USV =X$to circumvent the inversion of the covariance matrix but this will again fail as $S$ can only have $n<<p$ elements and thus won't be able to have enough information to whiten all $p$ columns. Oct 31, 2022 at 23:36
• I realised that my comment need expansion so I had to write an answer... So sleepy... Nov 1, 2022 at 3:21