Is it a better idea to use Principal Component Analysis as a preprocessing step for Linear Regression ? I have seen some people using PCA for reducing dimensions of the data ( i.e., losing dimension(s) that do not have any variation ) and then applying Linear Regression on it, though I do not know how effective such a procedure would be. Hence, I went forward and implemented it.
Training pattern before applying PCA
Ignoring a dimension that carries a variance of < 0.1 and transforming it results in 4 dimensional data. Now, the training pattern of Linear Regression (LR) after applying the transformation.
As you could notice, the validation error is not as great as when I did LR before applying PCA. Could someone explain what gives raise to this behaviour ? In my opinion, this should not lead to such worse performance, after all the dimensional I lost only contributes an overall of 0.1 variance though ?
Any help in enabling deeper insight on this would be appreciated.