1
$\begingroup$

I am trying to build a regression model that has continuous and categorical predictors. Furthermore, the continuous variables suffer from collinearity. My understanding is that PCA can handle multicollinearity but not categorical variables.

My questions are as follows (Question 1 is my main question):

  1. Can I do a PCA to get the principle component (PC) scores of the continuous variables and then take both the PC scores and categorical variables to build a mixed datatype regression model? Or is there a concern with this approach (and is there a better method)?

  2. When should I do a transform to the data, if needed? Before PCA or after PCA (i.e., before the regression model)? I ask because many of my variables are geometric measurements of length, area, and volume (i.e., this is a power function with 1, 2 and 3 for the power).

$\endgroup$
1
$\begingroup$

Multiple regression can "handle" multicollinearity; it's just that there will be vary large standard errors for coefficients of multicollinear variables. So you might not need PCA at all.

That said, there's also no need to omit categorical variables from the PCA. A categorical variable with k levels is transformed into (k-1) dummy variables, each coded 0/1, as is done silently by statistical linear regression software. These dummy variables can be standardized to 0 mean and unit variance just like the continuous variables for PCA. Whether that's useful for your application is another question, but it can be done.

I don't see a problem with separating out the categorical variables as you propose. You could use cross-validation to choose the number of principal components to include. But if you have multicollinearity that includes the categorical variables, then you have not accomplished what you set out to do in the first place.

All this leads me to suggest ridge regression for your application. It handles multicollinear variables together like PCA, because it effectively is a principal-components regression with differential weighting of components instead of all-or-none selection. As a result, the coefficients will tend to be more consistent among different samples from the same population than with standard multiple regression, while controlling for overfitting.

Whichever approach you choose, however, you need to work with variables that together are linearly related to your outcome variable. If that requires transformations, splines, or whatever then that should be done at the beginning. Note that PCA also involves linear relations among variables; transforming components after PCA would seem to be very hard to describe to others and seems at first thought to be prone to problems.

$\endgroup$
  • $\begingroup$ Thank you, EdM. Your response was very helpful. I was thinking the same with a post-PCA transform being hard to describe but wasn't sure if that would be problematic or not to a statistician. $\endgroup$ – Jon P May 4 '16 at 19:35
  • $\begingroup$ Maybe not to a statistician, but it would be to me, who is probably more like your target audience. $\endgroup$ – EdM May 4 '16 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.