Orthogonality in PCA vectors Why should the second PCA vector i.e. vector with largest variance in reduced subspace be orthogonal to the first PCA vector?
 A: From a simple geometric point of view: If the second eigenvector was not orthogonal to the first, then either the first eigenvector would not account for as much variation as possible, or the second eigenvector would not account for as much variation as possible.
Try drawing an ellipse and try drawing vectors through the ellipse where either the first or the second vector doesn't correspond to the principal axis of the ellipse.
Now try to imagine what the data points contained inside the ellipse would look like when projected unto those two vectors where one of the is slightly offset. You'll notice that in order to maximize variance along the direction of the vectors you've drawn, you HAVE to draw them orthogonally and you have to draw the first vector through the major axis of the ellipse.
A: The first (second) PCA direction is given by the eigenvector of the covariance matrix of the data corresponding to the largest (second largest) eigenvalue, see also this question.
The covariance matrix is symmetric and eigenvectors of (real) symmetric matrices are orthogonal, which gives the result.
