0
$\begingroup$

I am comparing the perfomance of PCA regression (i.e. regression where original regressors are replaced by few their pr. components) to that of regression with the original regressors in which I added some white noise (mean=0,std=1) to the regressor with the highest correlation coefficient r with the response variable.

I expect a decrease in R squared value when adding noise to one of the independent variables as this increases the unexplained variance in my model.

But I have read about PCA regression as a method able to denoise data as the noise component (or part of them) usually stays in the last components.

So far even if do not use these last components the value of R-squared is more or less the same when not lower.

My question:

  • Is it really possible to denoise independent variables with PCA-regression?
  • If so, how should I proceed?
$\endgroup$
  • 1
    $\begingroup$ "PCA regression" can eliminate that noise indeed if you conceptualize (independently of the existence of Y variable) your p observed independent variables as few m constructs plus p-m dimensional noise (error) orthogonal to those m ones. In your exercise of adding noise you added it right to the strongest predictor of Y; that action is incomparable in theory and consequences with "PCA regression". $\endgroup$ – ttnphns Sep 22 '17 at 10:31
  • $\begingroup$ Not sure to have understood your answer. "you added it right to the strongest predictor of Y; that action is incomparable in theory and consequences with PCA regression" Should I add noise to all the predictors and then take the components with better correlation with Y and highest eigenvalues? @ ttnphns $\endgroup$ – gis20 Sep 22 '17 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.