I am looking to tease out the significance and contribution of a particular variable to 2 different continuous responses. I have 7 continuous variables I know to be influential on the two responses (which have been considered by the literature). I also have a categorical variable that I wish to analyze to determine which response, if any, it is significant to and the degree of that significance. I have a small sample size, 18 observations. So I worry about power and error by using multiple linear regression on the 2 responses with 8 predictors. There is also a degree of colinearity between the continuous variables. I was wondering about the drawbacks or error in using principal component analysis on the 7 continuous variables, and extracting components to use for multiple regression analysis. Then in addition to those PC's, adding the categorical variable in the MLR. Therefore the equation would look like this: $$ Y = \beta_0 + \beta_1*PC Score + \beta_2*Categorical $$ The reason I am suggesting this approach is because I want to find what contributions above and beyond the 7 other variables the categorical variable contributes and control for confounding. Any suggestions/insight or alternative approaches would be awesome.

  • $\begingroup$ Seems like a reasonable approach to me. $\endgroup$ – Peter Flom Jan 23 '15 at 11:11

This is a reasonable approach, and you probably want to do something to reduce the dimensionality of your study. Since you asked about drawbacks and errors, here are some nightmare scenarios, with suggested remedies.

  • Using the first PC score assumes that the predictors impact Y through a linear relationship, as in a 1 factor model. If that's not true, then the 1PC model won't capture everything that's going on with the predictors.
  • If the omitted information is correlated with the categorical variable, then you might have created a confounding problem with interpretation of the categorical variable.You want to see what the categorical contributes beyond "the 7", but your transformed analysis will only show what the categorical contributes beyond a linear combination of the 7.
  • PCA, like all variance based methods, likes big sample sizes (i.e. more than 18 subjects).

And here are some possible fixes.

  • Presumably, you won't use the PC approach unless the first PC actually does explain a lot of the variation.
  • The first PC often just delivers the mean of the variables (especially if you standardized them first). So you could always just take the mean of your predictors and go from there. I'm saying this given that the small sample size makes it difficult to produce reliable estimates of the coefficients for the PC scores.
  • If you had a larger sample, I would recommend something like lasso regression to see if some of the predictors could be turfed out. As it is, you might try an informal "best subsets" -- at least to check if any of the 1 or 2 variable regressions are non-significant. Unless you have serious subject matter reasons for including all of them, perhaps some of them could be dropped. That, too, would be a way of reducing the complexity of the problem.
| cite | improve this answer | |
  • $\begingroup$ +1, but that "The first PC [...] delivers the mean of the variables" is only true if all the variables are positively (and similarly) correlated. $\endgroup$ – amoeba Jan 23 '15 at 18:14
  • $\begingroup$ That is correct. In many applications, where the variables all represent some sort of positive input, the first PC often ends up being a sort of indicator of overall size, accomplishment, preference ... whatever it is. Not much is lost by taking the mean and skipping the PC in that case. That's also why the second PC is often more interesting. $\endgroup$ – Placidia Jan 23 '15 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.