Say I have a set of data points in $\mathbb{R}^d$. At each time step, I add a point to the set, or remove one. I then fit a multivariate normal distribution to the points in the set, and compute the entropy of this distribution. I also run some other computations, which involve evaluating the density at various points.
The naive way to do this would be to re-compute the mean and covariance matrix from scratch at each time step. Of course, this would be computationally wasteful. An improvement would be to maintain running sums, and use these to efficiently compute the mean and covariance matrix at each time step. But, computing the entropy and the density requires a decomposition of the covariance matrix (e.g. eigendecomposition, cholesky decompositon, etc.). The 'improved' approach would still require re-computing this decomposition at each time step.
What's needed is a way to iteratively update the decomposition of the covariance matrix, given the data point added to or removed from the set. Can you recommend a good way to do this?
If the data points had constant mean, this problem could be formulated in terms of rank one updates. There are some papers describing this kind of approach. Unfortunately, this isn't the case; adding/removing a point changes both the mean and covariance matrix.