How to properly "subtract" a known covariance component from a sample covariance? regression

Question

I have a situation where observed random variables $X_i$ are the sum of two independent (but unobserved) variables, $$X_i = S_i + N_i,$$ (e.g. what you observe is a random signal plus random noise).

I have samples of the $X_i$ and I also know the distribution of $N_i$ (it's multivariate Gaussian with mean 0 and known, but non-diagonal, covariance). How do I estimate the covariance among the $S_i$?

My naive attempt:
Since $S_i$ and $N_i$ are independent, $\mathrm{Cov}(X) = \mathrm{Cov}(S) + \mathrm{Cov}(N)$. I can take my samples of $X_i$, calculate the sample covariance, and subtract off the known covariance matrix of $N$, yielding an estimate of the covariance of $S$. The problem is that with a finite sample size, the sample covariance of $X$ is not a perfect measurement of $\mathrm{Cov}(X)$, and when I do the subtraction I end up with an estimate of $\mathrm{Cov}(S)$ that is not positive-definite.

This seems like it's probably a straightforward statistics problem that I don't know the terminology to search for. Is there a simple, reliable way to "subtract off" a known component from a measured sample covariance that yields a valid (i.e. positive definite) covariance estimate?

(I would be happy to assume $S_i$ is multivariate Gaussian if it helps. Unfortunately, the dimension of the variables is about $1500$ so it seems like a tall order to parameterize the covariance matrix of $S$ with $1500^2 \approx 2$ million free parameters and try to fit them simultaneously.)

Answer 1

[Too long for a comment]

The issue here is that the space of (semi) positive definite matrices is not a vector space. A first solution would be to compute $$ \inf_{\Sigma \in \mathcal{S}_+} D\Big( \hat \Sigma_X, \Sigma + \Sigma_N\big), $$ where $\mathcal{S}_+$ is the set of sdp matrices and D is any metric of your choice between sdp matrices.

There also seems to be an important feature of the problem that you do not explain in details, i.e., you are facing a high-dimensional problem and do not say anything about the sample size. It is a well-known fact that empirical covariance matrices in high dimension have very biased spectra. The largest eigenvalues are too large and the smallest too small. You might want to recourse to other estimators; see ``A well-conditioned estimator for large-dimensional covariance matrices'' by Ledoit and Wolf for instance. A keyword here ist shrinkage estimators. It might be that you do not run into your problem with another estimator of $\Sigma_X$

Answer 2

Consider an additive approach. You know the distribution of $N$. Construct a distribution for $S$ with increasing levels of detail (constraints) until you are satisfied with your model for $S$, and thus $X = S + N$.

In a maximum entropy framework, match the "features" (functions of X) you care about. Matching the diagonal of $~\textrm{Cov}\left(X\right)~$ is easy. Lets say you have $J$ additional features that matter to you, you need to match $1500 + J$ constraints, not $1500^2$.

See Dempster 1972, Zhu, Wu, and Mumford 1998, and my discussion Keane 2019.