I was thinking about the difference between PCA and Linear regression in the simple two dimensional case.
In particular, I’m wondering which is better in capturing the information? By Gauss-Markov, simple linear regression is the best model (of the linear form), so why would we, in dimensionality reduction from 2D to 1D use PCA at all?
Let X1 and X2 be features. Why don’t we just use X1 to predict X2 and use X1 as our reduced dimension? Is there any advantage to using PCA in this 2D example?
I do not worry about the case with more than 2-dimensions because then the context matters as you cannot directly project more than a single dimension
Phrased another way, in the 2D case, which is greater, the sum of squared vertical distances from the regression line or the sum of squared perpendicular distances from the PC1 axis?