This is really a side project of mine ... while writing on a paper on something totally different! I read (part of ) the excellent paper
"FINITE SAMPLE APPROXIMATION RESULTS FOR PRINCIPAL COMPONENT ANALYSIS: A MATRIX PERTURBATION APPROACH" BY BOAZ NADLER
This shows among other things that misbehavior of PCA is connected to "crossings of eigenvalues". If you are interested in estimating the eigenvector corresponding to the largest (model) eigenvalue, then if the largest sample eigenvalue frequently comes from a smaller (model) eigenvalue, which is a "crossing", the corresponding sample eigenvector do not correspond to what we try to estimate, and PCA breaks down.
So, given a sample, it would be interesting to estimate (for example, using the bootstrap) the probability of crossing. How can we do that? Write the sample covariance matrix as $S$ and a covariance matrix corresponding to a perturbation of the sample $S^*$. (The perturbation could come from the bootstrap, or, conceivably, some other source). The problem now is that the eigensolutions of $S$ and $S^*$, computed by some algorithm, not necessarily are "the same", so we cannot pair of eigenvalues from $S$ with eigenvalues from $S^*$. ¿What does this mean? Write $t \mapsto t S + (1-t) S^* = S_t$. To be able to "pair of" eigenvalues we need an eigensolution of $S_t$ that depends continuously on $t$! The usual algorithms do not give us that. So, my question is: Anybody knows about such algorithms? any other related ideas? papers?
Since I did'nt get any answers here, I asked a related questio on Mathoverflow: https://mathoverflow.net/questions/116123/how-to-find-define-eigenvectors-as-a-continuous-function-of-matrix which got good answers.