Averaging between two sparse PCA models

Question

I am trying to construct a sparse PCA model for a large amount of data that I obtain as batches over time. This means e.g. that I can not draw the gloabal mean beforehand, since the the data is simply not there. I also do not want to store the whole data and recalculate it.

The idea is to keep only the vectors and the scores of the old PCA. With a new batch coming, I'd like to calculate a PCA model for the batch, and then recompute the new eigenvectors and scores basing on the information contained in the old model and the batch, with regard to the number of points that were used to create the two.

I do understand that there is a classical method for achieving this, and it should be even simpler than the PCA for point data, but I just don't know where to seek for a reference. Will be thankful for any help.

Addendum: More generally, the problem is how to calculate PCA not for point clouds, but rather for collections of weighted voluminous objects, e.g. $m$-dimensional ellipsoids in $n\ge m$-dimensional space, under the condition that the ellipsoid sizes may not be neglected.

Answer 1

After some time, I came up with what may be a solution for the problem.

Setup: The data is given by a small collection of $m$-dimensional ellipsoid hyperplane segments in an $n\ge m$-dimensional space - they are just collections of means, eigenvectors and singular values. Each ellipsoid $E_i$ has weight $w_i$. The task is to find $m$ principal components of the data. The ellipsoids' dimensions and positions are large in comparison to the expected singular values for the whole data.

Idea: Form covariance matrices from the ellipsoid data.

Approach: The core of the idea is to replace sums in the PCA with the integrals. Very generally, given a measurable subset $X\subset \Bbb R^n$ and a weight function $w(x)\colon X \to \Bbb R$, we may center $X$ by computing the mean $\mathbf{y} = \int_X w(y)dy$. Assuming the data to be centered, the covariance matrix for $X$ is computed by $Cov(X) = \int_X w(y)yy^Tdy$.

With $X$ now being a finite union of weighted ellipses, centering the data is trivial. The covariance matrix is then given by

$Cov(X) = \sum_i c_i\int_{E_i}(x_i + y)(x_i + y)^Tdy$

where $E_i$ is the $i$'th ellipsis centered at the origin, while $x_i$ is its center and $c_i = \frac{w_i}{\int_{E_i}dy}$ where $w_i$ is the "weight" of the ellipsis (in my case - the number of points from which it was obtained by a honest PCA procedure). Now the most pleasant part: since $E_i$ is symmetric with respect to the origin and $x_i$ is constant, we have $\int_{E_i}x_iy^Tdy=\int_{E_i}yx_i^Tdy=0$. Thus

$Cov(X) = \sum_i (w_ix_ix_i^T + c_i\int_{E_i} yy^Tdy) = \sum_iw_i(x_ix_i^T + Cov(E_i))$

The covariance matrix of $E_i$ is given by conditions. Basically, this means, that we have established a new covariance matrix, for which principal components may be calculated.

Problem is that, when the data is multidimensional, while we are interested in just a few main components, we still need need to make PCA for the matrix rather then infer them by some nice "averaging" of the eigenvalues for each ellipsis. Though the whole data needs not be stored any more and covariance matrix needs not be recalculated.

For myself, I have found another solution that suits for my particular use-case. But I'll just let this answer stay here, perhaps it may be inspirational.