Skip to main content
2 of 2
added 567 characters in body
Karel Macek
  • 2.8k
  • 15
  • 26

enter image description here The version without PCA is depicted first. You multiply the input by the vector of weights that define the normal of the hyperplane.

The version with PCA is depicted as second. You multiply the input by the PCA projection first, then multiply it by the (much shorter) weight vector.

The equivalence of LR+PCA and LR only happens if the black parts in the first and second cases equal each other.

An intuition can illustrate the situation: with PCA, you chose the hyperplane from limited options only (dashed). It is equivalent if it is equal to the version without PCA. enter image description here

This can happen if and only if there is a random vector $z\in \mathbb{R}^n$ that predicts a random variable $y$ that depends on $z$, and this dependence corresponds to logistic regression. If there is a matrix $M\in\mathbb{R}^{n\times m}$ with $n<m$ and this matrix corresponds to an inverse PCA transformation, then $x=Mz$ is something that has equivalent link to $y$.

In the real world, this can happen if $x$ contains a lot of irrelevant information whereas $z$ contains the relevant information for predicting $y$. However, it must follow the PCA properties.

Karel Macek
  • 2.8k
  • 15
  • 26