20
$\begingroup$

Let's say I have $N$ covariates in my regression model, and they explain 95% of the variation of the target set, i.e. $r^2=0.95$. If there are multicollinearity between these covariates and PCA is performed to reduce the dimensionality. If the principal components explains, say 80% of the variation (as opposed to 95%), then I have incurred some loss in the accuracy of my model.

Effectively, if PCA solves the issue of multicollinearity at the cost of accuracy, is there any benefit to it, other than the fact it can speed up model training and can reduce collinear covariates into statistically independent and robust variables?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Cross-posted at datascience.stackexchange.com/q/106887/55122 $\endgroup$
    – Ben Reiniger
    Commented Jan 9, 2022 at 21:32
  • 1
    $\begingroup$ You could and should also compare what happens if you reduce the number of covariates included in the model, period. In many fields, 95% explained would be a sign not so much of predictive success as of uncritical overfitting or near tautology (e.g. yesterday's temperature is generally a good predictor of today's temperature). Exceptions could arise e.g. in experimental fields and in some special set-ups. $\endgroup$
    – Nick Cox
    Commented Jan 10, 2022 at 10:56
  • 4
    $\begingroup$ PCA of covariates is one of many statistical topics that divides experienced people all the way from "would never do it on principle" to "often do it in practice". The one uncontested detail is that PCs are guaranteed uncorrelated with each other. But at best they just redistribute the information in the covariates and a regression with PCs as covariates isn't always easy to interpret or to compare with another regression with PCs as covariates. How important that is depends on the project. $\endgroup$
    – Nick Cox
    Commented Jan 10, 2022 at 10:59
  • $\begingroup$ PCA is a coordinate transformation, i.e., fitting data to n-ellipsoid. Data reduction is just an artefact of this procedure if we apply ranking. I am not entirely sure IF your statement is universally correct "Effectively, if PCA solves the issue of multicollinearity at the cost of accuracy" . There are works reporting otherwise here, here. $\endgroup$
    – patagonicus
    Commented Jan 10, 2022 at 18:38

6 Answers 6

Reset to default
33
$\begingroup$

Your question is implicitly assuming that reducing explained variation is necessarily a bad thing. Recall that $R^2$ is defined as: $$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} $$ where $SS_{res} = \sum_{i}{(y_i - \hat{y})^2}$ is a residual sum of squares and $SS_{tot} = \sum_{i}{(y_i - \bar{y})^2}$ is a total sum of squares. You can easily get an $R^2 = 1$ (i.e. $SS_{res} = 0$) by fitting a line that passes through all of the (training) points (though this, in general, requires more flexible model as opposed to a simple linear regression, as noted by Eric), which is a perfect example of overfitting. So reducing explained variation isn't necessarily bad as it could result in a better performance on unseen (test) data. PCA can be a good preprocessing technique if there are reasons to believe that the dataset has an intrinsic lower-dimensional structure.

Improve this answer
$\endgroup$
4
  • 7
    $\begingroup$ Thank you for the clarification. This also ties very much into bias-variance tradeoff. $\endgroup$
    – Jay Ekosanmi
    Commented Jan 9, 2022 at 21:52
  • $\begingroup$ "fitting a line that passes through all of the (training) points " Wouldn't "curve" be more appropriate than "line"? $\endgroup$
    – Acccumulation
    Commented Jan 11, 2022 at 3:24
  • 2
    $\begingroup$ @Acccumulation: Typically when we are talking about fitting a line, we mean linear coefficients in (possibly non-linear) terms, each of which is a function of measurable data, which can produce a plot that isn't straight. "Curve" is much more general and permits hidden state variables and a plot that doesn't correspond to a single-valued function. $\endgroup$
    – Ben Voigt
    Commented Jan 11, 2022 at 16:58
  • $\begingroup$ What's a better maxim: R² gains beyond when the test-set accuracy peaks are definitely overfit, before that they might be a mix of good or overfit? $\endgroup$
    – smci
    Commented Jan 13, 2022 at 2:21
10
$\begingroup$

In your question there is an implicit assumption about the regressor being linear.
In case it is linear your assertion is correct.

But for the case of non linear regressor you may think on the dimensionality reduction step as a feature extraction.
In that case it has a very important role in order to get good results.
It might reduce the noise, it might assist with the learning, etc...

Improve this answer
$\endgroup$
10
$\begingroup$

If the principal components explains, say 80% of the variation (as opposed to 95%), then I have incurred some loss in the accuracy of my model.

Performing PCA does not reduce the accuracy of the model. The principal components, when you use all of them, should also explain the 95%. It is the reduction of the dimensionality which reduces the explained variation.

So this is a matter of model selection and finding models with fewer parameters. The role of PCA is to do this model selection by redefining the parameter space in order to find a small number of components that explain a large amount of variation.

Improve this answer
$\endgroup$
10
$\begingroup$

Data reduction (unsupervised learning) is not always used because of any hope of wonderful performance, but rather out of necessity. When one has the "too many variables too few observations" problem, the primarily alternatives are penalized maximum likelihood estimation (ridge regression, lasso, elastic net, etc.) or data reduction. Data reduction, which as a side benefit deals well with collinearity, can be more interpretable, and works in any predictive context. Data reduction is IMHO much preferred over variable selection, because in the majority of problems variable selection yields a result that is too random/unstable. The spirit of data reduction is this: Estimate the model complexity that can be supported by your available sample size. Reduce the dimensionality (in a way that is completely masked to Y) and fit a single model whose number of parameters (that are estimated against Y) is supported by the effective sample size.

When using variable clustering or sparse principal components, one represents groups of variables with scores. Sometimes an entire group can be dropped. This procedure is not distorted by collinearities.

Improve this answer
$\endgroup$
2
  • $\begingroup$ You mention pros of PCA and cons of variable selection. What is the tradeoff here? What are cons of PCAs and advantages of variable selection? When would you prefer the latter? $\endgroup$
    – Christian Aichinger
    Commented Jan 11, 2022 at 4:37
  • $\begingroup$ Variable selection has little chance of finding the "right" variables and is very unstable. Data reduction is more stable. In my simulations, incomplete principal components regression predicts better than variable selection. For interpretability I use variable clustering followed by PCA or I use sparse PCA. I'm leaning towards the latter of late. Much more information may be found in Regression Modeling Strategies. $\endgroup$
    – Frank Harrell
    Commented Jan 11, 2022 at 13:10
3
$\begingroup$

Take a simple example of computing seasonal adjustment factor for months across a set of years for a company's sales. Assume there is no linear trend except if years are associated with an inflationary period. Note: In reality, one would work a log transform of the data which assumes a constant percent change relationship across time.

Collapsing the month data across years produces good results by month if inflationary periods are rare. If you happen to guess that the year seasonality is in a non-inflationary period, you have the best estimates with the best error estimates. So, the dimensionality reduction (ignoring years) is clearly best.

However, if it turns out that you are in an inflationary periods, not so good monthly seasonal adjustment. However, a year model may capture the inflation trend and produce better results.

So which model to use, collapsed or full?

One approach is to estimate the probability that you could be an inflationary period based on history,

Next, what is the operational cost associated with having an average error of X in a months' seasonality.

Knowing the difference in cost by month due to modeling error for collapsed vs full for inflationary versus non-inflationary, and the associated probability of each case, one can make a decision that produces the lowest expected cost.

This assumes that this exercise repeats over time and starting parameters estimate are good estimates.

So, the specific answer relates to the nature of the data, model specification/estimation precision and associated knowledge relating to error cost estimates.

Improve this answer
$\endgroup$
3
$\begingroup$

Let me give a quick chime in. I'm a data analyst for DNA methylation data. I have an awesome dataset that is about 1,000 people x methylation measure in over 3 million locations each x 3 points in time. That's nearly 10 billion data points.

If I want to analyze this data set... well, let's say some processes can take several days to weeks to run.

Alternatively, I can do a data reduction with PCA/UMAP and work with a much much smaller dataset that will give me fairly accurate and generalizable results within minutes. So even if my explained variation is lower, it can really make sense.

Something else that is worth considering is what are your results being used for? If my output is being used to make an important life-or-death decision then I really want to minimize any and all errors. If my output is going to inform further future research then I have a larger margin of error to work with.

Just a quick example, let's say my analysis yields the top 20 drugs that could be effective to treat a certain cancer. If this is going to a patient, I want drugs 1-3 to represent the best, 2nd best, and 3rd best treatment options respectively. Here the result has to be super precise, with very little error.

If, however, the goal is to try these out in cell cultures, then I don't really care if the top result happens to be the 3rd best or the 1st one. As long as my top 25% contains drugs that are more likely to work I don't really care about the order. (A lot of assumptions in this statement but just to exemplify).

To summarize, dimensionality reduction can also help with computation time for big data analysis. This is highly dependant on what the output will be used for.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.