Whitening a dataset with fewer observations than variables I have a k x n dataset where k equals the number of variables and n equals the number of observations per variable. I know these data are correlated and I would like to whiten them with the ordinary whitening transformation. Inconveniently, k outnumbers n by far, so that when estimating the k x k covariance matrix it will not be invertible. To get around this problem, I estimated the covariance matrix using optimal shrinkage, but the obtained covariance matrix is not suited for whitening anymore. 
I'd be grateful for ideas on how to whiten these data.
 A: I am doing the working assumption that by "whitening", we mean "sphereing" in the sense that data is uncorrelated and of equal variance along each dimension (not just to remove the mean and rescale the variance to 1 per dimension). I also note that the even when we are in $n > p$ setting the whitening matrix itself is non-unique; the three main ones are PCA whitening, ZCA whitening (sometimes called Mahalanobis whitening) and Cholesky whitening. The PCA is likely the most commonly used but the other are equally correct.
While anti-climatic I don't think there is a simple solution to this when $p > n$, $n$ being the number of observations and $p$ being the number of variables/dimensions/coordinates. That is because the information for the rescaling all the variables while accounting for the covariances simply isn't there.
The simplest sphering can be done via the Cholesky decomposition, we effectively take the covariance matrix $C$, invert it to get $C^{-1}$, decomposite as $C^{-1}=LL^T$ and use $L^T$ as the "sphering matrix". In the case of $C$ being rank-deficient though, the inverse doesn't exist so our immediate avenue is stopped. As suggested we can try using the Moore–Penrose pseudo-inverse to get our pseudo-inverse $C^{+}$ such that $CC^{+}C=C$. That is unfortunately a dead-end because the Cholesky decomposition still needs a positive definite (PD) matrix (we can actually get some little extra mileage  out of the $LDL^T$ decomposition as it is willing to play ball with positive semi-definite (PSD) matrices but as soon as $p - n> 1$ things break down again). The pseudo-inverse $C^{+}$ is not PD so Cholesky whitening is a no-go.
Next we try ZCA; ZCA actually has two equivalent formulations, one based on the inverse of the covariance matrix and one based on SVD ($USV^T = X$), the sphering matrices for those are $C^{-\frac{1}{2}}$ (that's why it is called Mahalanobis) and $VS^{-1}V^T$ respectively. The case of $C^{-\frac{1}{2}}$ is a no-go as described above - even if we get the inverse getting the square root of it is impossible as that matrix is not PD. The SVD one is the more promising as we do not invert anything directly but then we are faced with another problem: despite it's best intentions, the diagonal matrix $S$ is $n \times n$, not $p \times p$. That means that simply put, we cannot rescale $p$ varibles  with $n$  variances if $n < p$ ($V$ will be $n\times p$ so that's fine). So that again doesn't work.
Finally we have the PCA variant; it uses  the eigendecomposition of the covarinace matrix $C = U\Lambda U^T$ and the sphrering matrix is simply $\Lambda^{-\frac{1}{2}}U^T$. Now, if $C$ is rank-deficient we will still get $p$ eigenvalues which  is a step forward from the SVD above but those eigen-values are numerically zero as $C$ is (at best) PSD. Effectively the eigendecomposition comes back deficient by itself. Granted we can set small/numerically zero eigenvalues to something like $\epsilon$ and get the whole pipeline working  but the resulting sphrered data will be have $p-n$ columns with near $\epsilon$ variance and likely near  $\epsilon$ covariance with the other columns. The reason for that is that by "fixing" the numerically zero eigenvalues we zero-ed the variance along these  components/coordinates.
To recap, we cannot directly whiten/make spherical a rank-deficient matrix using any of the standard sphrering methodologies. Some methodologies do not compute at all (e.g. Cholesky whitening) and some truncate the dimensions that have variabillity (e.g. PCA  whitening). I used Optimal whitening and decorrelation by Kessy et al. (2018) as my standard reference. Do note that we can still using ridge or LASSO regression  if we want to use that data further, we just need to standardise the columns  in that case which is by itself much easier to do.
A: You didn't tell us why you want to whiten the data, but, however, you could simply use the generalized inverse of the sample covariance matrix the same way you uses its inverse (when it exists). It should have the same effect. (I can add details if wanted)
