Relationship Between Logistic Regression and Principal Component Analysis

Question

I have been into discussion about the usefulness of applied PCA before Logistic Regression for some classification task, the claim is as follow:

In some cases, Applying PCA and Logistic Regression together leads to almost the same separating hyperplane as just Logistic Regression alone (PCA + LR ∼ LR ). In such cases there is a specific relation R between the projection hyperplane of PCA and separating hyperplane of LR.

What is the relation R? and why R is necessary and sufficient for PCA + LR ∼ LR?

Answer 1

The version without PCA is depicted first. You multiply the input by the vector of weights that define the normal of the hyperplane.

The version with PCA is depicted as second. You multiply the input by the PCA projection first, then multiply it by the (much shorter) weight vector.

The equivalence of LR+PCA and LR only happens if the black parts in the first and second cases equal each other.

An intuition can illustrate the situation: with PCA, you chose the hyperplane from limited options only (dashed). It is equivalent if it is equal to the version without PCA.

This can happen if and only if there is a random vector $z\in \mathbb{R}^n$ that predicts a random variable $y$ that depends on $z$, and this dependence corresponds to logistic regression. If there is a matrix $M\in\mathbb{R}^{n\times m}$ with $n<m$ and this matrix corresponds to an inverse PCA transformation, then $x=Mz$ is something that has equivalent link to $y$.

In the real world, this can happen if $x$ contains a lot of irrelevant information whereas $z$ contains the relevant information for predicting $y$. However, it must follow the PCA properties.