In his popular course, Andrew Ng mentions using PCA to speed up supervised learning (Lecture 14.7). The basic idea is dimensionality reduction, wherein the extremely high-dimensional input features $\{x^{(1)},\ldots, x^{(k)}\} \in R^{N}$ are mapped to $\{z^{(1)},\ldots, z^{(k)}\} \in R^{M}$, where $M \ll N$.
Intuitively, the argument that training for lower-dimensional data should be quicker feels correct, but I could not find a good explanation or proof for this intuition.
My specific question is about the influence of de-correlation of input features that PCA performs towards speeding up learning. My intuition is that decorrelated features should be quicker to learn since the covariance matrix of the transformed $z^{i}$ features is a diagonal matrix.
For example, in a neural network, the weights for learning correlation between transformed features $z^{i}$ do not require much training since they are already de-correlated.
In other words, suppose I choose not to reduce dimensions after PCA (let's just say the eigenvalues are comparable), my supervised training would still be faster due to de-correlated transformed features. Is this intuition correct?
