Curse of dimensionality: How PCA improves my model?

Question

After having read about the curse of dimensionality, I have been looking into a filtered version of the Superconductivity dataset.

The number of dimensions is 81, so initially I thought that reducing the dimensions using PCA could help to improve the prediction power. (The majority of the articles I have read about the curse of dimensionality speak about dimensions greater than 10) Surprisingly, I found the opposite:

# Using caret library and a loop to iterate 
# over the number principal components to use:
for (pca_component_i in seq(5, 80, by = 5))
    {
        fitControl <- trainControl(
            preProcOptions=list(pcaComp = 
              pca_component_i),
            method = "repeatedcv",
            number = 4,
            repeats = 2,
            summaryFunction=defaultSummary,
            verboseIter = FALSE
        )
    
model1 <- train(train_X, train_y, 
            method = "glm", 
            trControl = fitControl,  
            preProc = c("pca"),
            metric =  "RMSE"
        )
print(RMSE(predict(model1, newdata=test_X), 
      test_y))
    }
        
    [1] 20.42709
    [1] 18.36957
    [1] 15.0958
    [1] 11.05231
    [1] 2.58729
    [1] 1.250649
    [1] 0.3855323
    [1] 0.2942177
    [1] 0.1621667
    [1] 0.1147847
    [1] 0.03312724
    [1] 0.02567603
    [1] 0.01053921
    [1] 0.007634504
    [1] 0.001618915
    [1] 7.672395e-06

On the other hand, it also looks pretty obvious behavior cause more principal components explain more variance. So,

• What am I misunderstanding?
• Is there any possible situation where reducing the dimensionality of the data lead to a increase of the prediction power?

Answer 1

With over 20,000 cases in the linked data set, your 81 predictors probably aren't getting you into trouble with "the curse of dimensionality" for this application. From pages 265 - 266 of "An Introduction to Statistical Learning," second edition (ISLR) on this topic:

In general, adding additional signal features that are truly associated with the response will improve the fitted model, in the sense of leading to a reduction in test set error. However, adding noise features that are not truly associated with the response will lead to a deterioration in the fitted model, and consequently an increased test set error. This is because noise features increase the dimensionality of the problem, exacerbating the risk of overfitting (since noise features may be assigned nonzero coefficients due to chance associations with the response on the training set) without any potential upside in terms of improved test set error.

The associated example (Figure 6.24) had 100 observations, with 20 features truly associated with outcome and additional random features in LASSO modeling. In this situation they started to get into trouble when the number of candidate predictors got above 20 or so, or a case/predictor ratio of about 5.

In contrast, you have about 250 cases per predictor. One typically gets into problems when this ratio is down close to the single digits or, as can happen for example in bioinformatic analyses, you have more potential predictors than cases so that you must reduce the effective dimensionality of the predictor set for regression. As the example from ISLR demonstrates, even LASSO for penalization might not help with a very large predictor set if most predictors are unassociated with outcome.

Answer 2

Higher order PC components need not be the most important features.

PCA only helps you when the higher variation components are meaningful in explaining the variation in the $Y$ variable.

• Indeed, Sometimes PCA helps you in finding meaningful correlation patterns in the features and this could indicate some relevant principle that might align with the $Y$ variable.
• But, the pattern can be something else and may not need to explain $Y$.

see for instance the image from this question correlation of features and target in predicting red wine quality in machine learning

In the figure above, the difference between the groups is very subtle and does not correspond with the large variability in the parameters.

Within group variability

The situation of a low explanation power by higher order principal components occurs when some parameters are correlated and have a large variance, which creates the PCA components, but the differences in the variable $Y$ of interest is perpendicular to this.

This may happen if the features can have a large variability given the same value for the parameter of interest (e.g. given the same group or given the same critical temperature). Especially when the variation within the group follows some relationships (and is correlated), then the PC1 component will be more relating to within-group variation than between-group variation. In the example above this relationship is between sugar and alcohol which are negatively correlated (because sugar is turned into alcohol)

Example plot with your data

In a lot of physical/chemical experiments parameters are varied more or less along curves (defined by some physical law with a bit of variability). An example is below using your data set where we only focus on two features (for simplicity of plotting). The experimenter has the ability to change atomic mass and density by changing the composition, but they are very much correlated, and not every combination is possible. The variation is mostly along a curve.

The PC1 or other higher-order PC components are gonna be aligned tangent to those curves, but the relevant parameter of interest might be varying perpendicular to this.

The dataset is especially having this issue because it has the same parameter repeated several times in a slightly different way, like geometric mean and arithmetic mean. Below you see this for a plot of the atomic mass. You see this green region in the middle (the high temperature conductors), which is stretched out along a line. This probably relates to components that are a particular ideal mixture of small and large atomic mass atoms such that the geometric mean is along a line about 0.75 times the arithmetic mean (the pure compositions are along the line geometric mean = arithmetic mean).

Answer 3

It is a mistake to go into your problem thinking that fewer parameters is better. Remember the bias-variance decomposition. You are reducing the parameter count to lower the variance, but that doesn’t necessarily come with a small increase to the bias. It is totally possible that, by omitting relevant features, you bias the model too much to result in a decrease in MSE.

This answer of mine does not address PCA, but it does address the idea of decreasing the number of parameters in an attempt to improve performance and why that need not help.