2
$\begingroup$

I have a linear regression problem with about 120 predictors and I tried to remove a number of predictors from it. First I tried to remove multi-collinearities by calculating the variance inflation factor. This left me with about 20 different (hopefully not collinear anymore) predictors. Then I used a PCA to reduce dimensionality even further. Because the predictors' variances are very different to one another I used the correlation matrix for this.

I can get the 'final' data when I multiply the eigenvectors with the largest eigenvalues with my original data, right?

In the end I want to find out which original predictors are left and how I can recover the 'new' original data. But for some reason I am not able to recover correct numbers.

Improve this question
$\endgroup$
2
  • $\begingroup$ Which software are you using? And the principal components are linear combinations of the original predictors, so it doesn't make much sense to ask which predictors are left. To obtain the new data, which I assume is the original data projected onto the first $k$ of the $p$ principal components, multiply the original data matrix by the matrix whose columns are the principal components. $\endgroup$
    – assumednormal
    Jul 4, 2012 at 10:43
  • $\begingroup$ This doesn't sound like a sensible approach to subset selection to me. $\endgroup$
    – Michael R. Chernick
    Jul 4, 2012 at 10:49

1 Answer 1

Reset to default
4
$\begingroup$

PCA does not get rid of any of the variables, although some may be very unimportant. Suppose you use the first three components from your PCA, each of these is a linear combination of the 20 variables you put into it.

If you want to be able to interpret the importance of the original variables in the regression, I don't think PCA is the way to go. I would consider one of the penalized regression methods such as LASSO or LAR

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ So PCA helps me to reduce dimensionality, so I can let's say save the data more efficiently, but in this case I can't remove any of the original counters. $\endgroup$
    – david
    Jul 4, 2012 at 11:24
  • 1
    $\begingroup$ However, you can look at how much each original variate contributes to the scores (= new variate values) for different cases. If your original data was standardized, the PC loadings will tell you that. If not, you need to multiply the loadings element-wise with the original data (i.e. stop before the summation of the matrix multiplication you need to obtain the scores). $\endgroup$
    – cbeleites unhappy with SX
    Jul 4, 2012 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.