0
$\begingroup$

Hoping somebody can help me. I cannot find an example that 'finishes' a problem. I run a proc princomp in SAS. I have hundreds of variables but used four for the purpose of an example. I understand the idea of an eigenvalue and proportion as it relates to variability explained of the dependent variable. lets assume that the first two eigenvalues explain 85% of the dependent variable. here are the resulting eigenvectors. prin1 prin2 prin3 prin 4 w -0.55 0.18 0.63 0.51 x 0.52 0.25 0.71 -0.41 y 0.65 0.03 -0.05 0.76 z -0.05 0.95 -0.3 -0.02

now, assume I want to run a regression. do the prin's become components that are in essence variables? i.e new_indep_var1=-.55*w+.52*x+.65*y-.05*z, etc etc. in essence having 4 variables to run in a regression model with whatever dependent variable I want, call it v? thanks

$\endgroup$
0
$\begingroup$

Yes, you should think of the transformed data as a new dataset. It would have three columns (2 features and the target) and do regression as you usually would.

You have to ask yourself however, why you're doing this. The only legitimate reason I can see why you might do this, is to avoid over-fitting, because you think including all 4 original variables will make you more susceptible to over-fitting. Even then, there are probably better ways of doing this (e.g. include all features but use regularisation).

Why do I say this? Well remember that your new features, let's call them $z_{1}$ and $z_{2}$, are as you say, both just a linear combination of your old features. Thus if you find that $y = \alpha z_{1} + \beta z_{2} +\eta$, then you're actually saying

$y = \alpha \left(ax_{1} + bx_{2} + cx_{3} + dx_{4} \right) + \beta \left(ex_{1} + fx_{2} + gx_{3} +h x_{4}\right) + \eta$

which is the same as saying

$y = a'x_{1} + b'x_{2} + c'x_{3} + d'x_{4}+\eta$

Your result won't be the same as if you did linear regression with all 4 parameters, as you only effectively have two degrees of freedom, but you're still learning a multilinear functional dependence of your target on 4 features.

$\endgroup$
  • $\begingroup$ there are actually dozens of variables (steel industry). I only listed four to more easily convey example. thank you for response $\endgroup$ – GKJohn Oct 30 '18 at 23:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.