One component in PCA is always the mean vector in two-dimensions but not three I've been testing PCA via SVD to decompose a simple time series data matrix, $X$.  I have two signals $x_1(t)$ and $x_2(t)$ in a data matrix where $M$ rows represents each timepoint sample and each column represents $x_1$ and $x_2$.
The mean signal, $\hat{x}$, is defined as the mean along the row axis (average of $x_1$ and $x_2$ along each timepoint).  I normalize each column of $X$ by subtracting its mean and dividing by the standard deviation.
When I use [U S V] = svd(Xz) in matlab, regardless of how the variables are distributed (whether they are correlated or uncorrelated), one of the columns of the right singular matrix, V, always points in the same direction (to a multiplicative constant) as the mean vector $(1/2, 1/2)$.  But when add an additional third time series, this is never the case (where the mean vector is $(1/3, 1/3, 1/3)$.
Because I normalize the standard deviation for each vector, it does make sense that the direction of most variance given by PCA would be the diagonal 45 degree line.  But if both variables $x_1$ and $x_2$ are independent gaussians, couldn't the PCA direction be any direction since the distribution is radially symmetric?
MATLAB Code:
s = RandStream('mcg16807', 'Seed', 0);
RandStream.setDefaultStream(s);

G = zeros(1000,2);
G(:,1) = 40*randn(1000,1)-100;
G(:,2) = tan(G(:,1)) + randn(1000,1);
X = G - repmat(mean(G),[size(G,1), 1]);
Xz = X./repmat(std(G),[size(G,1), 1]);
[U S V] = svd(Xz);
G_mean = mean(Xz,2);
corrcoef(G_mean, U(:,1))
corrcoef(G_mean, U(:,2))

 A: The answer is elaboration of my comment, because you asked. The core action of PCA is singular-value decomposition of data matrix X. It is the same thing as eigen-decomposition of square symmetrical matrix X'X: both actions will leave you the same matrix of eigenvectors (which within SVD you call V matrix). This matrix of eigenvectors is the matrix of orthogonal rotation of old axes (columns of X) into new axes (principal components), and its elements are cosines between these and those.
When columns of X are standardized then X'X is a correlation matrix (multiplied by N-1; this multiplication will have no effect on the matrix of eigenvectors). Thus, PCA of standardized data columns is eigen-decomposition of correlation matrix between the data columns.
You have 2 standardized data columns. Whenever correlation between them is not exactly zero you will always get eigenvector matrix which is the 45 degree rotation. If the correlation is exactly zero, no rotation is needed and eigenvector matrix is {1,0;0,1}.
