Out-of-sample reconstruction error with PCA when space dimensionality is larger than number of sample points I've got a question and I have done several experiments in R, yet couldn't figure out why.
The question is for a data set of N*D, N for number of data points and D for dimension, the maximum number of principal component is max(N, D).
Then I discovered a fact that, when N is larger than D, by using D principal components, any old data(data within the data set) could be reconstructed absolutely the same as it was, so can any new data, the result of
((new_data - mean) %*% loadings %*% t(loadings) + mean) ### code in R
is the same as new_data.
However, when D is larger than N, by using N principal components, reconstruction of data within the data set is still precise, that of the new data differs from the originals. I just can't figure out why. Can you please explain or give some hint?
 A: I think you have some mistaken notions about PCA. The maximum number of principal component is D. Principal components are orthogonal to each other, so once you have D of them, you span the entire space. You wouldn't be able to find a D+1 principle component in a space of D dimensions, regardless of N>D.
The first part is as expected. When you express your N data points in D principal components, it is a simple change of basis. Change of basis should be reversible, if the new basis is full ranked, which it is in this case.
If D>N, your N datapoints lie within a smaller subspace of at most N dimensions. By using N principal components, you should be able to span that N subspace. The PCA would only be able to find N principal components. If your new data, e.g. point N+1, lies outside of the subspace spanned by the N principal components, you don't have the basis to represent the point N+1. Thus you get reconstruction error. If you do PCA again including the new data points, you will again get zero reconstruction error on your data.
A: @mt0 provided a nice answer (+1), the only thing missing there is an illustration. So in an attempt to win the Worst CrossValidated Illustration Contest here is my drawing captured by a laptop camera:

Here in both cases $D=2$. On the left $N>D$, so there are two principal components, and new points (open circles) can be perfectly represented in the new coordinates (given by two PCs). On the right there are only 2 data points, so $N-1<D$ and there is only one principal component to be estimated. Now the new points (again, open circles) can only be approximated by their projections on this first principal components and there is nonzero reconstruction error depicted by dashed lines.
