Potential Misunderstanding of PCA I am playing around with PCA by plotting a regular 256x256 grid of points, randomly rotating the grid, and computing the principal axes. Originally, I thought I would be able to recover the amount that I had rotated the grid using the principal axes, but it seems that is not the case, and it makes sense to me that on a 90 degree or 45 degree interval the covariance would be 0, but for some arbitrary rotation, like say 75 degrees, I still get the basis as [[-1, 0],[0, 1]] as in the attached image. I must be misunderstanding something about what these principal axes mean, can anyone clear up my confusion?
 A: PCA tries to select orthogonal directions which capture maximum variance. I suspect the following is happening in your case,
If your data is spatially uniformly distributed (that is if input data is simply set of all points point on the grid), then probably it can't find any best direction. In other words, if there's almost equal variation along each direction, then there's no way to rotate the grid to select directions which cover maximum variance.
A: Hssay provides a good clue, that there is not a 'best' direction. However, it is not so much because of the uniformly distributed data. Uniform distribution is neither sufficient nor necessary. So, I wish to add to this refining on that situation. 
The data does not need to be evenly distributed for this effect (missing 'best' direction) to occur, and also the effect may not occur in all evenly distributed data sets (for instance if you used a rectangular grid instead of a square grid).
The effect that you have is indeed a variance equal in all directions (if you imagine the data points on a circle rather than a square grid, then you might better get the intuitive idea). 
You get this when you have an algebraic multiplicity of the eigenvalues (equal to geometric multiplicity since the correlation matrix is symmetric ie diagonalizable). Then the variance in two (perpendicular) directions is equal. So any other combination of these directions will result in equal variance. Say we have variables vectors $\vec{x}$ and $\vec{y}$, then
$$Var(a\vec{x}+b\vec{x}) = a^2 Var(\vec{x}) + b^2 Var(\vec{y}) + 2ab  Cov(\vec{x},\vec{y}) $$ 
and because $x$ and $y$ are perpendicular (covariance is zero) and $a$ and $b$ are from a rotation matrix ($a^2+b^2=1$) and the variance of the two vectors are equal ($Var(\vec{x})=Var(\vec{y})$) you will get that with any rotation the variance remains the same.
So it is not the uniform distribution (which is not a necessary condition, and even not always true as well, as in 'homogeneous $\neq$ isotropic'). It is the variance being equal in (at least) two perpendicular directions, which is a sufficient (and neccesary) condition. You can get this also in non-homogeneous distributions.
