2
$\begingroup$

I'm using PCA for a while, but recently I read about reconstruction error which I cannot understand...

For example let's consider dataset consisting 5 variables: $X_1, X_2, X_3, X_4, X_5$. Additionally I obtained two variables $Z_1$ and $Z_2$ by using principal components analysis using two components which are linear combinations of $X_i$, $i = 1,2,3,4,5$. What in this case I should perceive as a reconstruction error?

The most logical thing for me would be:

  1. Calculate RMSE of $Z_1$ with $X_1, X_2, X_3, X_4$ and $X_5$. Output is then 5 numbers.

  2. Calculate RMSE of $Z_2$ with $X_1, X_2, X_3, X_4$ and $X_5$. Output is also consisting 5 numbers.

What I would say is that reconstruction error is mean of those 10 numbers. Am I correct?

$\endgroup$
2

1 Answer 1

4
$\begingroup$

I have seen this term "reconstruction error" in the context of PCA before.

I will skim over most of the details of PCA, but I recommend you become adequately familiar with diagonalization and SVD in the context of standard linear algebra (if you are not already).

One of the things that PCA provides is a transformation of the data in the form of a change in basis. This change in basis is invertible.

When performing PCA for the purpose of dimensionality reduction, a subset of the principal components are chosen either for exploratory data analysis or as machine learning features.

Taking only a subset of your principal components, and the corresponding inverse change in basis, you can convert your transformed variables back to the original space of your data.

If you kept all of the principal components in the inverse transformation, you should perfectly reproduce your data. In reality there can be some floating point error, an issue with your computer keeping track of only a finite number of decimal places, but this is not what we mean by "reconstruction error" in this context.

Rather, if you apply the inverse transformation using only a subset of your principal components, you probability will not reproduce the original dataset perfectly.

Computing a loss function, such as RMSE or similar functions, between the original data and the reconstruction of the data yields a number that we call "reconstruction error".


Here is an illustration of the process, if it helps your understanding.

enter image description here

Code for the diagram, if you want to use or modify it.

from graphviz import Digraph

g = Digraph('stack')

g.node('Reconstruction\nError',shape='square')

g.edge('Transformed\nData',
       'Data',
       label='Inverse\nChange\nof\nBasis')
g.edge('Data',
       'Transformed\nData',
       label='Change\nof\nBasis')
g.edge('Transformed\nData',
       'Dimensionality\nReduced\nTransformed\nData',
       label='Select\nComponents')
g.edge('Dimensionality\nReduced\nTransformed\nData',
       'Lossy\nReconstructed\nData',
       label='Inverse\nChange\nof\nBasis')
g.edge('Lossy\nReconstructed\nData',
       'Reconstruction\nError',
       label='RMSE',
       style='dashed')
g.edge('Data',
       'Reconstruction\nError',
       label='RMSE',
       style='dashed')
g.view()
$\endgroup$
3
  • 1
    $\begingroup$ (+1) Nice use of graphviz. $\endgroup$ Commented Dec 4, 2021 at 11:26
  • $\begingroup$ Can the non-linearity of data impact reconstruction error? $\endgroup$
    – hafiz031
    Commented Jan 9, 2023 at 4:42
  • 1
    $\begingroup$ @hafiz031 Sure. If your data is sampled uniformly from a circle in Cartesian coordinates then you can compress the data only so much with a linear transformation. If you change that in Cartesian coordinates circle to polar coordinates then your circle maps to a line. Lines are nicely compressed with PCA. $\endgroup$
    – Galen
    Commented Jan 9, 2023 at 5:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.