Dear Cross Validated Users,

I am addressing myself to you with a question for which I couldn't find an answer despite intensive googling. I want to run a regression with a multitude of independent variables. The independent variables can be grouped into groups of highly correlated variables. I want to be able to interpret the results, thus I would like to apply some kind of dimension reduction method to the individual variable groups. Intuitively I would apply a PCA to each group and use the resulting principal component for each group as an explanatory variable in the final regression. I would thus get the 'optimal' weighting of variables within each group and be able to interpret the coefficients of the artificially (by dimension reduction) created summarizing variables. Is there some method which does what I want?

I am grateful for every comment!

Background: I am measuring different network statistics of a dynamic network with different window sizes from which I want to estimate different dependent variables. Unfortunately there is no way to tell which window size is the correct one. So I am computing the network statistics over a number of different window sizes. I want to create some kind of surrogate explanatory variables for each network statistic over all available window sizes.


PCA sounds like a good thing to do within the groups of correlated predictor variables. You could use the rule-of-thumb for each PCA run and set the number of PCs required from each run based on the number of eigenvalues greater than unity, $\lambda_j>1$. For example, your first PCA run has 50 predictors, hence a $50 \times 50$ correlation matrix $\mathbf{R}$, and maybe 7 eigenvalues are greater than 1, hence #PC=7. The second PCA run has 30 predictors, so a $30 \times 30$ $\mathbf{R}$ correlation matrix for which maybe 3 eigenvalues were greater than one, hence #PC=3. For these two PC runs you now have 10 PCs which are uncorrelated predictors.

Unfortunately, a lot of beginners(students) want to throw everything into a single model to solve everything in one run. However, you should think about breaking up your models and dependent variables, where you might regress each $y$ on each set of PCs from the intended PCs (relevant predictors which were reduced via PCA). Don't initially plan on using one unifying model. Instead, plan on breaking up a large regression into small regression models. If you had only one $y$ or several multivariate $y$'s, you could regress them on all of the PCs from all the PCA runs, but if you have different $y$-variables associated with each set of (original) predictors, then run those regressions using the $y$ on the PCs you extracted via PCA.

Don't make one large unifying model -- try to break things up into many models, because they will have their own peculiarities and different goodness-of-fit characteristics.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.