Is it a better idea to use Principal Component Analysis as a preprocessing step for Linear Regression ? I have seen some people using PCA for reducing dimensions of the data ( i.e., losing dimension(s) that do not have any variation ) and then applying Linear Regression on it, though I do not know how effective such a procedure would be. Hence, I went forward and implemented it.

Training pattern before applying PCA

Training pattern before applying PCA

Ignoring a dimension that carries a variance of < 0.1 and transforming it results in 4 dimensional data. Now, the training pattern of Linear Regression (LR) after applying the transformation.

Training pattern after applying PCA

As you could notice, the validation error is not as great as when I did LR before applying PCA. Could someone explain what gives raise to this behaviour ? In my opinion, this should not lead to such worse performance, after all the dimensional I lost only contributes an overall of 0.1 variance though ?

Any help in enabling deeper insight on this would be appreciated.

  • $\begingroup$ What is shown on the x-axis? Why would the validation error be so much lower than the training error (top figure)? I'd say the blue line should be associated with validation, not training. Then all would make sense ;). $\endgroup$
    – Michael M
    Apr 25 '18 at 12:31
  • $\begingroup$ x, and y axes correspond to epochs, and Mean Squared Error respectively. I am unsure what you mean by 'blue line should be associated with validation, not training', you mean just swap the colours ? Yet, focusing just on the pattern, is there anything you could perhaps infer ? $\endgroup$
    – VM_AI
    Apr 25 '18 at 12:39
  • 1
    $\begingroup$ Usually, training accuracy is better than validation accuracy and definitively not much worse. So I was wondering if you mislabeled the curves? $\endgroup$
    – Michael M
    Apr 25 '18 at 13:14
  • $\begingroup$ Agree that validation error is typically larger than training error, so the plot seems unusual. PCA does not eliminate low variance covariates from the original data, but rather it identifies dimensions of largest variance and rotates the axes to correspond. If you have colinearity on your data, you'll be able to reduce dimensions. Feeding that to another model assumes that your direction of greatest variance is correlated with your response, which is not guaranteed. The data could be noisy or biased in some fashion. $\endgroup$
    – KirkD_CO
    Apr 25 '18 at 13:17
  • $\begingroup$ I can't comment futher as the first plot shows a problem unrelated to PCA. $\endgroup$
    – Michael M
    Apr 25 '18 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.