I would like some clarification as to how principal component analysis mitigates the Curse of Dimensionality problem. My particular interest is in curbing overfitting in my modelling, or more specifically parameter count. If I use all 30 of my features I will have a model with 30 parameters: this is too large a number for my sample size, overfit is almost guaranteed. I am told that I should rather build my model with the first 3 principal components of my feature set and thus have only a 3 parameter model, and apparently mitigate my overfitting problem. But then I have computed 30x30 elements for my eigenvector matrix and 3 parameters for my model, I have fitted 900+3 parameters to the data. Now I have gone from a model with a maximum parameter count of 30 to a model with 903 parameters. How have I evaded the Curse of Dimensionality? It is really not obvious to me. An additional issue is the high variance of the elements of the eigenvector matrix, I have noted that relatively small changes to the feature data cause considerable variation in these elements, sometimes even changing signs. They are more unstable than the parameters of the model that I am trying to fit.
-
$\begingroup$ Things aren't quite as bad as you make out. The eigenvector matrix is orthogonal and so is parameterized with $30+29+\cdots+1=30(31)/2$ real parameters. That's still a lot. But notice that by estimating coefficients of the first three PCs you are really using "just" $30+29+28=77$ parameters total and they are highly redundant. After all, when all the smoke clears your model still is a linear combination of $30$ features, so in a roundabout way you have arrived at employing at most $30$ parameters. $\endgroup$– whuber ♦Commented Mar 28, 2022 at 15:48
-
1$\begingroup$ Firstly thanks for the correction on the unique element count on the eigenvector matrix, I will look into that. But you have to compute all 30(31)/2 parameters before I select the 77 I am going to use for my principal components, the parameters are not computed independently. No I am fitting 30(31)/2 + 3 parameters to the data. I don't understand how you arrive at : " you have arrived at employing at most 30 parameters. " Please clarify. We have added another optimization to the process with a high parameter count, and have high variance. $\endgroup$– Andrew BeavenCommented Mar 28, 2022 at 16:44
-
$\begingroup$ OK I appreciate Bjorn's point that the PCA optimization is not fitted to the response, so won't be as damaging to generalization error as I might have thought. $\endgroup$– Andrew BeavenCommented Mar 28, 2022 at 16:58
-
$\begingroup$ Another question I have about the PCA optimization is related to correlation. So if variables have a high correlation then they will fit easily into one of the PCs. Great if they all have the same - say - positive correlation with the response. But what if the PCA optimization flips the sign of a feature (that is negatively correlated with the response) and adds it to that PC? That then would reduce predictive power. $\endgroup$– Andrew BeavenCommented Mar 28, 2022 at 17:20
-
$\begingroup$ It doesn't matter if you compute a billion parameters along the way, if you wind up with $30.$ That's just an algorithmic detail with no statistical implications at all. As far as sign flipping goes, that's meaningless in PCA. See stats.stackexchange.com/questions/34396 or related threads about signs in PCA. (Eigenvectors represent linear subspaces, not directions, and can be freely negated to suit your tastes.) $\endgroup$– whuber ♦Commented Mar 28, 2022 at 17:26
2 Answers
In a way, PCA does not use the outcome you are trying to model/predict, i.e. it is an unsupervised technique. From that perspective, its parameters are not parameters that get trained in your supervised model. Of course, using PCA for dimensionality reduction is not in any way guaranteed to preserve "the signal" for the outcome of interest that may be in the data (see e.g. this previous question for a discussion). I.e. it may well be preferable to select the most important variables based on subject-matter expertise, if there is a decent amount of prior knowledge. There's of course also other techniques/alternatives to PCA (e.g. various variants of PCA, UMAP, t-SNE, training a denoising autoencoder on the features and so on).
However, a lot may also depend on your goals. Are you trying to interpret the model coefficients (if so, PCA does make that harder), are you trying to create a prediction model that is meant to achieve a certain level of performance (if so, interpretability of PCA may be less of a concern, but it may also be even more of a problem to be working with too little data), or are you trying to do something else?
-
2$\begingroup$ I don't follow your remark that "PCA does make that harder," because PCA is used only as a procedure to obtain the same $30$ coefficients that would otherwise be estimated and they continue to have the same interpretations. $\endgroup$– whuber ♦Commented Mar 28, 2022 at 15:50
-
$\begingroup$ Good point. I guess with that extra step, it ends up being reasonably interpretable, although I'm not entirely sure how I get SEs and so on for the individual components of a principal component (do you know a solution for that??). I was thinking only of "principal component #1" has a regression coefficient of -1.78 (SE 2.5), which is a bit of a mess. $\endgroup$– BjörnCommented Mar 28, 2022 at 19:49
-
$\begingroup$ It depends on whether you want SEs conditional on the PCA or unconditionally. For the latter, a parametric bootstrap would be attractive. For the former, it's a simple computation because the $30$ coefficients are linear combinations of the three estimated coefficients and you know their variance-covariance matrix. Note, though, that the variance-covariance matrix of all $30$ coefficients will have rank at most $3:$ that is, there will be very strong dependencies among them. $\endgroup$– whuber ♦Commented Mar 28, 2022 at 21:42
-
$\begingroup$ I guess conditional would be the easy way out, but would not really reflect the full uncertainty, would it? So, I'd guess ideally conditionally. I guess you could also do a simple bootstrap and average coefficients (setting them to zero for bootstrap samples, where the variable is not part of the first 3 principle components etc). $\endgroup$– BjörnCommented Mar 28, 2022 at 23:06
-
$\begingroup$ Agreed about not reflecting the full uncertainty. But a nonparametric bootstrap is a risky and uncertain thing to do when the amount of data is relatively small. That's why a parametric bootstrap deserves first consideration. It's unclear whether averaging coefficients would be any improvement on the proposed PCA regression; but the bootstrap distributions of those coefficients could give useful insight into the estimation uncertainty. $\endgroup$– whuber ♦Commented Mar 29, 2022 at 14:09
But then I have computed 30x30 elements for my eigenvector matrix and 3 parameters for my model, I have fitted 900+3 parameters to the data.
The possible solutions for the parameters relating to the features are strongly limited. You are effectively only fitting 3 parameters. Because the potential solutions $\hat{Y} = \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_{30} X_{30}$ lie in a 3d space.
