In my textbook, several methods such as Hierarchial, Forced Entry etc are discussed. It says:

"When predictors are all completely uncorrelated, the order of variable entry has very little effect on the parameters calculated; however, we rarely have uncorrelated predictors and so the method of predictor selection is crucial."

For Hierarchial specifically, it says:

"In hierarchical regression, predictors are selected based on past work and the experimenter decides in which order to enter the predictors into the model....... New predictors can be entered either all in one go, in a stepwise manner, or hierarchically (such that the new predictor suspected to be the most important is entered first)."

A regression equation is just an equation and the variables added as "+ beta * variable". How can the order even matter? Does it have anything to do with semi-partial correlation?


1 Answer 1


By "predictors are all completely uncorrelated", I think what they mean is the "sample correlations of the predictors are zero", because when the samples of predictors are centered, "sample correlations of the predictors are zero" is true if and only if "samples of the predictors are orthogonal to each other".

Proof: For two centered predictor samples $x_1$ and $x_2$ satisfying $\mu_{x_1} = \mu_{x_2}=0$, the sample correlation coefficient will be: $$ COR(x_1,x_2) = (x_1 - \mu_{x_1})^T(x_2 - \mu_{x_2})/(\sigma_{x_1}\sigma_{x_2}) = x_1^Tx_2/(\sigma_{x_1}\sigma_{x_2}) $$ So when $COR(x_1,x_2)=0$, that means $x_1^Tx_2=0$, thus $x_1$ and $x_2$ are orthogonal to each other; On the other hand, when $x_1$ and $x_2$ are orthogonal, $x_1^Tx_2=0$, so that $COR(x_1,x_2)=0$.

And when samples of the predictors are orthogonal to eacher, their coefficients in the multivartiate regression will be the same as the ones in the univariate regression, thus the order of the predictor doesn't matter. Say if $x_1 \perp x_2$, then when fitting $$ y = \beta_1^{(1)} x_1 + \beta_2^{(1)} x_2 \\ y = \beta_1^{(2)} x_1 \\ y= \beta_2^{(2)} x_2 $$ you will get $\beta_1^{(1)} = \beta_1^{(2)}$ and $\beta_2^{(1)}=\beta_2^{(2)}$.

This statement is no longer true when $x_1 \not\perp x_2$. For example when doing regression by successive orthogonalization, the predictors entered earlier will have an impact on the coefficient of the predictors entered later, if they are not orghogonal to each other.

For details on regression by successive orthogonalization and why orthogonal predictor samples matter, I recommend you read Elements of Statistical Learning, Hastie et al, chapter 3.2.3..


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.