How do PCA components change upon the addition of new data? How do the principal components from PCA change on addition of new data, i.e. $$\frac{d(\mathrm{PC}_1(x))}{dx} = ?$$
I am looking for any mathematical formulas and proofs (since it would be easy to understand).
 A: It's complicated, because the addition of one more data row can completely change all principal components and their eigenvalues.  However, we can do some analysis that sheds light on what is happening and allows for some general conclusions as well as providing a tool to understand any particular circumstance.

I am relying on the text of the question for its interpretation because the mathematical expression literally makes no sense.  Here is what I make of it.  Suppose the data are arranged, as usual, in an $n$ by $p$ array $\mathbb{X}$ with rows representing cases and columns representing variables (optionally standardized and usually centered--but for simplicity of exposition I will ignore those effects and implicitly assume that the new data row will not materially change the means or variances of any of the variables).  PCA is essentially the analysis of the eigensystem of the symmetric non-negative definite $p\times p$ matrix $\mathbb{A}=\mathbb{X}^\prime\mathbb{X}.$  Adjoining a new row $v$ to $\mathbb{X}$ changes $\mathbb{A}$ into $\mathbb{A}+v^\prime v.$
To understand the change induced by the new row, begin with the PCA solution for the original data.  It consists of an orthogonal basis of unit-length eigenvectors $(e_i)$ and their associated eigenvalues $\lambda_i$ in descending order so that $\lambda_1\ge\lambda_2\ge\cdots\ge\lambda_p\ge 0$.  In this basis $\mathbb{A}$ is diagonal with the values $\lambda_i$ on its main diagonal (and zeros off the diagonal).  Let $u$ be the vector $v$ expressed in this basis.  (That is, $u = u_1e_1+u_2e_2+\cdots+u_pe_p$ where $u_i = v\cdot e_i.$)
By means of row-reduction (or otherwise, depending on your tastes) it is possible to show that when $\mathbb{Y}$ is any diagonal matrix with values $y_i$ on the diagonal, then
$$\det(\mathbb{Y}+u^\prime u) = y_1y_2\cdots y_p\left(1 + \frac{u_1^2}{y_1} + \frac{u_2^2}{y_2} + \cdots + \frac{u_p^2}{y_p}\right).$$
(Because we have to contemplate the possibility of zero values for some of the $y_p$, this expression must be understood in the sense that $y_1y_2\cdots y_p / y_i = y_1\cdots y_{i-1}y_{i+1}\cdots y_p$.)
To find the new eigenvalues we must, by definition, solve the equation
$$\det(\mathbb{A} + u^\prime u - \lambda) = 0$$
for the eigenvalues $\lambda$.  Setting $\mathbb{Y}=\mathbb{A}-\lambda$ (which is still a diagonal matrix) and plugging it into the preceding formula gives the polynomial equation
$$p_{\mathbb{A},u}(\lambda) = (\lambda_1-\lambda)(\lambda_2-\lambda)\cdots (\lambda_p-\lambda)\left(1 + \sum_{i=1}^p\frac{u_i^2}{(\lambda_i-\lambda)}\right) = 0.$$
We can begin to draw some general conclusions:


*

*If $u_j=0$--that is, $u$ is orthogonal to the eigenvector $e_j$--then $\lambda_j$ remains an eigenvalue.  (This is geometrically obvious; it is equally obvious that $e_j$ remains the associated eigenvector.)

*When all the $u_j$ are sufficiently small, the eigenvalues will not change a lot.

*We should therefore be interested in how rapidly the eigenvalues will change with small perturbations $(u_j)$.  The rate at which $\lambda_k$ changes can be found by differentiating the characteristic polynomial $p_{\mathbb{A},u}(\lambda)$ with respect to $\lambda$ and evaluating it at $\lambda_k$.  I obtain
$$\eqalign{
\frac{\partial p_{\mathbb{A},u}(\lambda)}{\partial\lambda}\left(\lambda_k\right) 
&= (\lambda_1-\lambda_k)\cdots(\lambda_{k-1}-\lambda_k)(\lambda_{k+1}-\lambda_k)\cdots(\lambda_p-\lambda_k)\\
&\sum_j \left(1 + \sum_{j\ne k}\frac{u_j^2}{\lambda_j-\lambda_k}\right).
}$$
The derivative is infinite when $u_j\ne 0$ and $\lambda_j$ has multiplicity greater than $1$.  Otherwise, its size is determined by the magnitudes of the $u_j^2$ relative to the gaps $\lambda_j-\lambda_k$ in the sequence of eigenvalues.
There is much of a semi-quantitative nature we can glean from the last observation. For instance, it is evident that when the new row $u$ has a substantial component along a small principal direction, it will effect dramatic alterations in the smallest eigenvalues and should be expected completely to change the smallest principal directions.  In order to perturb one of the largest principal directions, though, $u$ must have a substantial component in a direction whose eigenvalue is of a comparable size.
One could go further to develop expressions for how the eigenvectors change as a function of the $(u_i)$, but they get sufficiently messy that it seems best to stop the exposition here and let interested readers carry out the calculations themselves, using the techniques illustrated here.
