2
$\begingroup$

Some time ago I read this great short paper on the relationship between 'correlation', ‘independence’ and ‘orthogonality’. In short, two vectors can be uncorrelated, they can be orthogonal, they can be both, or they can be neither.

I have noticed in differing regression textbooks the authors state different requirements for ensuring that coefficient values are not affected by the inclusion of additional independent variables. For example, in the textbook Introduction to Econometrics by Stock & Watson the authors state that if regressor variables are uncorrelated with one another then their coefficient estimates will be invariant with respect to the inclusion of additional independent variables. In the book Introduction to Linear Regression Analysis by Montgomery the author says that regressor variables must be orthogonal to one another in order to ensure that their coefficient estimates are not affected by one another. The professor who taught the class that used the second book says that when the book described orthogonality as being necessary, it was referring to the centered versions of the regressor variables. If that is true then it is in fact uncorrelated variables that have invariant coefficient estimates - at least with respect to the inclusion of additional variables, as the first book suggests.

When studying Design of Experiments one learns that the design matrices are specifically created in such a way that the coefficient estimates associated with a regressor variable are invariant with respect to the other variables used in the analysis. Interestingly, though, these design matrices don't just ensure that the IV's are uncorrelated (which is all that is necessary to achieve the result), they also ensure orthogonality.

My question is: is there a greater purpose for ensuring orthogonality in addition to 'uncorrelated-ness'? And/or is it just too difficult to ensure 'uncorrelated-ness' without also ensuring orthogonality?

$\endgroup$
2
$\begingroup$

Correlation and Dependent Parameter Estimates

I don't have the Stock and Watson book, but orthogonality is what you need to make sure your estimates of $\beta_i$ are independent.

Christensen (2002) derives the sampling distribution for $\hat{\boldsymbol{\beta}}$ in linear regression when all of the $\beta_i$'s are estimable as $$ \hat{\boldsymbol{\beta}} \sim \mathrm{N}\bigl( \boldsymbol{\beta}, \sigma^2 (\mathbf{X}'\mathbf{X})^{-1}\bigr)~. $$ Using the examples from the paper as our model matrix (so no intercept, and we're fitting $\beta_1$ and $\beta_2$) the uncorrelated but non-orthogonal matrix $$ \mathbf{X} = \left[\begin{array}{rr} 0 & 1 \\ 0 & 0 \\ 1 & 1 \\ 1 & 0 \end{array} \right] $$ gives us a covaraince matrix for $\hat{\boldsymbol{\beta}}$ proportional to $$ (\mathbf{X}'\mathbf{X})^{-1} = \left[\begin{array}{rr} 2 & -1 \\ -1 & 2 \end{array}\right]~ $$ where the non-zero off-diagonal entries indicate that estimates of $\beta_1$ will be different when we're also estimating $\beta_2$. I generated some gaussian noise in R and let $\beta_1 = \beta_2 = 1$, then I arrived at the following:

> cor(X)
     [,1] [,2]
[1,]    1    0
[2,]    0    1

> y
           [,1]
[1,]  0.4537310
[2,] -0.2468915
[3,]  1.0462463
[4,]  1.0854357

> coef(lm(y~X-1))
      X11       X12 
0.9211289 0.2894242 

> coef(lm(y~X[,1]-1))
 X1[, 1] 
1.065841 

Note that $\hat{\beta}_1 = 0.92$ under the full model but $\hat{\beta}_1 = 1.07$ under the restricted model.

Orthogonality in Designed Experiments

The nice thing about orthogonality is that it ensures $(\mathbf{X}'\mathbf{X})^{-1}$ is diagonal so there is no covariance between parameter estimates and our estimate of $\beta_1$ will not change depending on if $\beta_2$ is in the model or not.

If we are running a non-orthogonal experiment to determine which predictors have an effect on the response (I'd call this a screening experiment) then we don't know which $\beta_i$'s belong in the model and which do not. We'd have to recalculate a whole mess of things as we add and remove terms. In the orthogonal case we only have to make a calculation once, and there's no difference between our estimate of $\beta_i$ after $\beta_j$ was added and $\beta_i$ before $\beta_j$ was added. The interpretation is more clean, and back in the day the amount of hand calculations required for analysis added to the popularity of orthogonal designs.

But that's not to say orthogonality is required. In screening experiments there is often a sparsity of active effects - that is, most of the $\beta_i$'s are zero or negligible. Non-orthogonal designs like no-confounding designs (Jones and Montgomery 2010) and definitive screening designs have small parameter estimate covariances and allow us to investigate more factors or more complex models in (sometimes drastically) fewer runs than an equivalent orthogonal design. Step/stage-wise procedures can be used for the analysis.

Citations

Christensen, R. (2002). Plane Answers to Complex Questions. Springer.

Jones, B. and Montgomery, M. (2010). Alternatives to Resolution IV Screening Designs in 16 Runs. Int. Journal of Experimental Design and Process Optimisation. Vol 1 #4. 285-295.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.