In PCA, do the principal components beyond the first optimize any expression? Given a covariance matrix $\mathbf\Sigma$, the first principal component $u_1$ is the unit vector that maximizes variance $u_1'\mathbf\Sigma u_1$. Do there exist similar expressions that the first $k$ principal components optimize when taken together? In other words, what do we maximize/minimize when we take out these principal components greedily?
One thought is that the first $k$ principal components define a subspace that maximizes the sum of norms of the projected vectors. This is indeed true when we maximize variance while calculating the first principal component. However, I'm not sure if this intuition, or something else, holds in general.
 A: The first $k$ principal components minimize the squared reconstruction error. That is, we project the data onto the first $k$ principal components, then back into the original space to obtain a 'reconstruction' of the data. The first $k$ principal components are the vectors that minimize the sum of squared distances between each point and its reconstruction (the paper below mentions this point, among many other sources).
Among all sets of $k$ vectors, the first principal components do not maximize the sum of the variance of the data projected onto each vector. For example, in many cases we could increase the variance by making all vectors point near the direction of the first principal component. But, if we constrain the vectors to be orthogonal (as PCA does), then the first principal components do indeed have this property (e.g. see here).
Another interpretation is that the first $k$ principal components maximize the likelihood of a particular Gaussian latent variable model. See the following paper:

Tipping & Bishop (1999). Probabilistic principal component analysis.

