I'd like using a stepwise linear regression model, where I include predictor by predictor according to their significance. However, in contrast to the standard procedure, I'd like to use a set of predictor variables, which are not all independent from each another.

Intuitively, this is just like in linear algebra: (principle component analysis)

  • If I would use a set of independent predictor variables, these predictors would represent an orthogonal basis. The responds variable would be expressed as a linear combination of these. The coefficients would be the "projections" of the responds variable onto the basis vectors.
  • The same interpretation is applicable if I use dependent predictor variables. Since the basis is not orthogonal, this might complicate the interpretation of each coefficient, if I'd like to find the optimal parameter set for an experiment. However, in principle there is nothing wrong with the procedure.

Is this picture correct?


2 Answers 2


There is nothing wrong with the procedure, I believe. I don't think it's 'standard procedure' to have all predictors independent, though. Predictors are rarely independent (unless one uses e.g. components from a PCA as predictors).

Indeed, for interpretation, dependency between predictors matters, because coefficients represent the expected increase in the dependent variable, if an predictor increases by 1, while all other predictors remain constant. The latter may be unlikely with dependent predictors.

Dependence between predictors does make the coefficients / the parameter estimates less precise. But this will be reflected in their standard errors, which will be larger when correlation between predictors increases.


I can't explain in terms of linear Algebra but I suggest to try structural equation modelling. This technique is the combination of factor analysis and multiple regression analysis, and it is used to analyze the structural relationship between measured variables and latent constructs.

This method is preferred by the researcher because it estimates the multiple and interrelated dependence in a single analysis


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.