Does PCA provide advantage if all PC's are used? The question is basically stated in the title. Does principle component analysis even provide an advantage if all principle components are used later? In an application I am looking at PCA is conducted before a simple regression. The author claims that if all PC's are included in the regression, the result will be equivalent to OLS on the original feature set. Is this true?
 A: @gunes is correct (+1) in terms of unpenalized ordinary least squares models. There is one situation in which PCA might be considered to "provide an advantage when all PCs are used," however, even in linear regression modeling.
Principal components regression (PCR) selects only a subset of PCs, an all-or-none 1/0 weighting of the components, as James et al explain in Section 6.3.1 of ISLR. Chapter 6 also covers ridge regression as a "shrinkage" or penalization method. James et al then compare these approaches (page 236):

One can even think of ridge regression as a continuous version of PCR

That is, ridge regression uses all the PCs but gives them different non-zero weights rather than the all-or-none PC selection used by PCR. Page 79 of ESL has more details. In that sense ridge regression does use all the PCs, just not equally. But that's not PCR in the sense of your question.
A: The PCs are just linear combinations of the original features. For example, if there are two features, $x$ and $y$, the features mapped on the PCs will be something like $f_1=\alpha_1 x+\beta_1 y$, and $f_2=\alpha_2x+\beta_2y$. So, it's just a change of axes.
In ordinary linear regression, the target variable is expressed in terms of linear combination of features, i.e. $y=ax+by+k$. Using the new features that are linear combinations of the old ones will generate an equivalent equation. For example, for two features, this would look like the following:$$\begin{align}y&=cf_1+df_2+k=c(\alpha_1x+\beta_1y)+d(\alpha_2x+\beta_2y)+k\\&=\underbrace{(c\alpha_1+d\alpha_2)}_ax + \underbrace{(c\beta_1+d\beta_2)}_by+k\end{align}$$
This is the case for OLS, but in general, does using all PCs have an advantage? Maybe. Having orthogonal axes may be paramount for the downstream analyses you'll perform depending on what you’re after, so generalizing this for all ML is not possible.
