In principal components analysis, the first $k$ principal components are the $k$ orthogonal directions with the maximum variance. In other words, the first principal component is chosen to be the direction of maximum variance, the second principal component is chosen to be the direction orthogonal to the first with the maximum variance, and so on.

Is there a similar interpretation for Factor Analysis? For example, I'm thinking that the first $k$ factors are the factors that best explain the off-diagonal components of the original correlation matrix (in the sense of, say, squared error between the original correlation matrix and the correlation matrix defined by the factors). Is this true (or is there something similar that we can say)?