0
$\begingroup$

I'm trying to verify my understanding of how to apply principal component analysis to a multiple regression. Here's my current process and understanding using Minitab:

Part 1: I already have my data in the necessary format. I run Principal Component analysis. In this analysis, I exclude any dependent variable I would use as the 'y' variable in a regression.

Question 1: If my purpose of using Principal Component Analysis is to use in regression, should I exclude "response" / "y" / "dependent" variables from the Principal Component Analysis?

Part 2: In Minitab, Principal Component Analysis outputs a matrix like this:

Eigenvalue  2.4876  1.8307  1.4844  1.0342  0.8997  0.6784  0.5411  0.4852  0.3776  0.1810
Proportion   0.249   0.183   0.148   0.103   0.090   0.068   0.054   0.049   0.038   0.018
Cumulative   0.249   0.432   0.580   0.684   0.774   0.842   0.896   0.944   0.982   1.000


Variable       PC1     PC2     PC3     PC4     PC5     PC6     PC7     PC8     PC9    PC10
Bid          0.083  -0.053  -0.252  -0.871   0.211  -0.092   0.002   0.311  -0.138   0.020
BrandLabel   0.129  -0.148   0.643  -0.158  -0.021   0.134   0.680  -0.022  -0.204  -0.012
Priority     0.309  -0.022   0.112  -0.118  -0.786  -0.489  -0.121   0.062   0.004   0.041
Impressions  0.327   0.353  -0.123  -0.045  -0.316   0.732  -0.028   0.152  -0.013   0.302
Clicks       0.344   0.449   0.095   0.112   0.387  -0.404   0.059  -0.060  -0.031   0.581
CTR          0.314  -0.311   0.400   0.089   0.206   0.119  -0.621   0.155  -0.412  -0.026
AvgCPC       0.381  -0.320  -0.275  -0.142   0.024   0.085   0.023  -0.801  -0.069   0.050
Cost         0.434   0.473   0.008   0.012   0.137  -0.041   0.024  -0.037   0.036  -0.751
AvgPos       0.238  -0.245  -0.501   0.392   0.000  -0.100   0.357   0.351  -0.466  -0.039
ImpShare     0.410  -0.407   0.013   0.082   0.159   0.032   0.070   0.286   0.738   0.033

From this output, according to the many YouTube videos I've watch on this subject, I should be most interested in the "components" that have Eigenvalues greater than 1.0. In this scenario, that means PC1 - PC4 which explains 68.4% of the variance. In said YouTube videos, some advice given recommended using only component values where the absolute values are greater than 0.30 as a rule.

Question 2: When multiplying my variables by the principal components columns, should I multiply ALL coefficients or should I be picking and choosing only certain coefficients over a certain threshold?

Part 3: I create Eig1 - Eig4 columns which multiply PC1 - PC4 by my actual columns. Ex: Eig1 = PC1 coefficients * my columns in the data, and so on. My question is about the VIF interpretation below. If Eig1 - Eig4 are not correlated, VIF would be 1.0. My values are very high which implies a lot of correlation (right?) or colinearity?

Question 3: Are VIF values in a regression that uses columns derived from Principal Component Analysis as the predictors supposed to be close to 1.0 ?

Regression Analysis: Conv versus Eig1, Eig2, Eig3, Eig4 

Analysis of Variance

Source           DF    Adj SS   Adj MS    F-Value  P-Value
Regression        4  17186270  4296568  261166.13    0.000
  Eig1            1     41903    41903    2547.04    0.000
  Eig2            1     37395    37395    2273.05    0.000
  Eig3            1     34299    34299    2084.83    0.000
  Eig4            1     16070    16070     976.84    0.000
Error          7705    126759       16
  Lack-of-Fit  3913    126759       32          *        *
  Pure Error   3792         0        0
Total          7709  17313029


Model Summary

      S    R-sq  R-sq(adj)  R-sq(pred)
4.05604  99.27%     99.27%      98.24%


Coefficients

Term         Coef  SE Coef  T-Value  P-Value         VIF
Constant   0.7117   0.0576    12.36    0.000
Eig1      -1.1450   0.0227   -50.47    0.000  2407950.86
Eig2       0.9689   0.0203    47.68    0.000  2279140.90
Eig3      -0.5021   0.0110   -45.66    0.000    72660.01
Eig4       0.6588   0.0211    31.25    0.000    34331.34


Regression Equation

Conv = 0.7117 - 1.1450 Eig1 + 0.9689 Eig2 - 0.5021 Eig3 + 0.6588 Eig4
$\endgroup$
  • $\begingroup$ Q0. Fundamentally Principal Components Regression has few advantages over OLS regression; it's real advantages are really computational. You should be sure that you want to do PCR. $\endgroup$ – user20637 Nov 10 '15 at 11:57
  • $\begingroup$ Q1. Depending on what you're trying to achieve, yes - exclude the dependent variable from the PCA. Q2. Calculation of PC scores needs centred and potentially scaled 'x' values. In Minitab there is no need to do it "manually", use the 'Storage...' option to specify the number of scores you want. That depends on Q0. Q3. Yes. VIFs for regression on PC scores (not eigenvalues) should be 1 $\endgroup$ – user20637 Nov 10 '15 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.