How to Compute the Reconstruction error in Principal Component Analysis at lower dimensions

Question

I have m examples and d features where m<<d. So I managed to compute the eigen value and corresponding its eigen vector ... I want to compute the reconstruction error for various value of principal components , lets say n principal componets (n eigen vector corresponsing to first max n eigen values)

So My original dataset has shape

(m, d)

If I take n principle components where n < d, the transformed matrix matrix would have shape

(m, n)

So Since the shapes mismatch I really wasnt able to find any way to compute the reconstruction error

Answer 1

Like you said, you cannot compute the error in that fashion because the dimensional space is not the same, you would need first to 'reconstruct' your new-reduced space to the old-bigger dimensional space and there you can compute the error as you said.

For the dimensionality reduction we do:

$T = XP$

for the reconstruction we do:

$\hat{X} = TP^{'}$

where $X$ is the original data, $T$ is the reduced space (formally called scores), $P$ are the loadings (PCs) and $\hat{X}$ is the reconstructed space.

Finally, we commonly express the error as:

$Error = (X - \hat{X})^{2}$

---

Edit:

How do we interpret Error:

Before, first we denoted $X$ as our matrix of original data $\mathbb{R}^{nxk}$ ($n$ data points with $k$ dimensions each), and then we moved to the reduced space $\hat{X} \in \mathbb{R}^{nxp}: p<k$

When we compute the error, we are computing the distance (norm) between the original ($p$ dimensional) and the proyected ($k$ dimensional) observation.