In his popular course, Andrew Ng mentions using PCA to speed up supervised learning (Lecture 14.7). The basic idea is dimensionality reduction, wherein the extremely high-dimensional input features $\{x^{(1)},\ldots, x^{(k)}\} \in R^{N}$ are mapped to $\{z^{(1)},\ldots, z^{(k)}\} \in R^{M}$, where $M << N$.

Intuitively, the argument that training for lower-dimensional data should be quicker feels correct, but I could not find a good explanation or proof for this intuition.

My specific question is about the influence of de-correlation of input features that PCA performs towards speeding up learning. My intuition is that decorrelated features should be quicker to learn since the covariance matrix of the transformed $z^{i}$ features is a diagonal matrix.

For example, in a neural network, the weights for learning correlation between transformed features $z^{i}$ do not require much training since they are already de-correlated.

In other words, suppose I choose not to reduce dimensions after PCA (let's just say the eigenvalues are comparable), my supervised training would still be faster due to de-correlated transformed features. Is this intuition correct?

In his popular course, Andrew Ng mentions using PCA to speed up supervised learning (Lecture 14.7). The basic idea is dimensionality reduction, wherein the extremely high-dimensional input features $\{x^{(1)},\ldots, x^{(k)}\} \in R^{N}$ are mapped to $\{z^{(1)},\ldots, z^{(k)}\} \in R^{M}$, where $M \ll N$.

Intuitively, the argument that training for lower-dimensional data should be quicker feels correct, but I could not find a good explanation or proof for this intuition.

My specific question is about the influence of de-correlation of input features that PCA performs towards speeding up learning. My intuition is that decorrelated features should be quicker to learn since the covariance matrix of the transformed $z^{i}$ features is a diagonal matrix.

For example, in a neural network, the weights for learning correlation between transformed features $z^{i}$ do not require much training since they are already de-correlated.

In other words, suppose I choose not to reduce dimensions after PCA (let's just say the eigenvalues are comparable), my supervised training would still be faster due to de-correlated transformed features. Is this intuition correct?

In his popular course, Andrew Ng mentions using PCA to speed up supervised learning (Lecture 14.7). The basic idea is dimensionality reduction, wherein the extremely high-dimensional input features $\{x^{(1)},\ldots, x^{(k)}\} \in R^{N}$ are mapped to $\{z^{(1)},\ldots, z^{(k)}\} \in R^{M}$, where $M << N$.

Part (A)

Intuitively, the argument that training for lower-dimensional data should be quicker feels correct, but I could not find a good explanation or proof for this intuition. Could you give me an explanation of why lower dimensional version of data is always quicker to train, i.e. does the model learning always converge faster?

Part (B)

I was also wondering about the influence of de-correlation of input features that PCA performs towards speeding up learning. My intuition is that decorrelated features should be quicker to learn since the covariance matrix of the transformed $z^{i}$ features is a diagonal matrix.

For example, in a neural network, the weights for learning correlation between transformed features $z^{i}$ do not require much training since they are already de-correlated.

In other words, suppose I choose not to reduce dimensions after PCA (let's just say the eigenvalues are comparable), my supervised training would still be faster due to de-correlated transformed features. Is this intuition correct?

Part (C)

Could you shed light on how PCA's de-correlation influences model convergence. Can I use early-stopping without the fear of under-fitting my data that has been transformed by PCA?

In his popular course, Andrew Ng mentions using PCA to speed up supervised learning (Lecture 14.7). The basic idea is dimensionality reduction, wherein the extremely high-dimensional input features $\{x^{(1)},\ldots, x^{(k)}\} \in R^{N}$ are mapped to $\{z^{(1)},\ldots, z^{(k)}\} \in R^{M}$, where $M << N$.

Intuitively, the argument that training for lower-dimensional data should be quicker feels correct, but I could not find a good explanation or proof for this intuition.

My specific question is about the influence of de-correlation of input features that PCA performs towards speeding up learning. My intuition is that decorrelated features should be quicker to learn since the covariance matrix of the transformed $z^{i}$ features is a diagonal matrix.

For example, in a neural network, the weights for learning correlation between transformed features $z^{i}$ do not require much training since they are already de-correlated.

In other words, suppose I choose not to reduce dimensions after PCA (let's just say the eigenvalues are comparable), my supervised training would still be faster due to de-correlated transformed features. Is this intuition correct?