Why do we get a new co-ordinate system when we dot product the transpose of eigen vectors with the transpose of a matrix

Question

I am working on implementing PCA on the MNIST dataset and have calculated the eigen vector and eig Values from the co-variance matrix. Now I want to have a new co-ordinate system represented by PC1 and PC2 just so I can project the datapoints on PC1. I have a code snippet where the new coordinate is created by taking the dot product of the transpose of eigen vectors with the transpose of the standardized data matrix.

eigValues, eigVectors = linalg.eigh(Covariance_mat, eigvals= (782, 783))
new_cordinates = np.matmul(eigVectors.T, standardize_mnist.T)
new_cordinates

Output:
``````
array([[-5.2264454 ,  6.03299601, -1.70581328, ...,  7.07627667,
    -4.34451279,  1.55912058],
   [-5.14047772, 19.29233234, -7.64450341, ...,  0.49539137,
     2.30724011, -4.80767022]])

Now, I am not sure about the intuition behind this dot product, which leads to a new 2d coordinate system.

I am very new to Machine Learning hence, could someone please help me understand this part.

Answer 1

Let the dimension be denoted by $d$. There are $d$ eigenvectors of dimension $d\times1$. In PCA< we choose $k<d$ of them an obtain an underrepresentation. Typically, the data matrix, $X$, as also in your case has dimension $n\times d$, where $n$ is number of data points in the dataset. So, each row is a sample. In order to convert a sample, say $x$ of dimension $1\times d$, to new coordinate system, we need project the sample onto the chosen set of eigenvectors, let them be denoted by $v_1,v_2...v_k$. Projection means calculating dot product, and projecting $x$ onto each eigenvector is equivalent to calculating the dot product $v_i^Tx^T$. So, the new coordinates for our sample will be the following: $$x_k=[v_1^Tx^T,\ v_2^Tx^T\dots v_k^Tx^T]=[v_1^T\dots v_k^T]x^T=V^Tx^T$$ If you do this for all the samples in the dataset, we have: $$X_k=V^T\begin{bmatrix}x_1^T\\x_2^T\\\vdots\\x_n^T\end{bmatrix}=V^TX^T$$