7
$\begingroup$

I am studying the well-known book Elements of Statistical Learning. When the multiple linear regression is described it uses the simple univariate regression as a building block, which makes sense to me. As far as I understand it uses orthogonality property of input vectors in order to split multivariate regression in simple independent regressions, and when inputs are not orthogonal, then those inputs are transformed in such a way that what remains is orthogonal. By orthogonal vectors I understand 2 vectors which have dot product equals with zero.

Now in the book is noted that

Orthogonal inputs occurs most often with balanced, designed experiments (where orthogonality is enforced), but almost never with observational data.

How can one enforce that? The only situation that I can imagine is when one would use binary 0/1 values for each possible nominal value. To be more clear: one could have a nominal column sex with labels: male and female. He can create two input columns, one called sex.male with value 1 when sex is male and 0 otherwise. The corresponding column sex.female would then have 1 if sex is female and 0 otherwise. These 2 numerical columns would be orthogonal. Is possible to do enforcing for continuous variables?

Improve this question
$\endgroup$
4
  • $\begingroup$ I dont think it can be enforced with real input data. You can of course do PCA in your data to transform it to orthogonal series. $\endgroup$
    – adam
    Jan 14, 2014 at 9:43
  • $\begingroup$ I am not sure what you mean by "enforcing". One can check whether the variables are orthogonal. But if they are not orthogonal then they aren't. Multiple regression is very often used when the input vectors are not orthogonal. $\endgroup$
    – Peter Flom
    Jan 14, 2014 at 11:27
  • $\begingroup$ @adam this is my belief also, however I am very fresh to statistics to really know that $\endgroup$
    – rapaio
    Jan 14, 2014 at 11:36
  • $\begingroup$ @PeterFlom by enforcing I understand a way to design the experiment that would cause the inputs to be orthogonal I am not sure however what authors means by enforcing. That was the question. If it exists such kind of design of an experiment in order to get orthogonal inputs. Solely my imagination produced only the scenario for nominal inputs, but for continuous I am not able to design such thing. $\endgroup$
    – rapaio
    Jan 14, 2014 at 11:42

1 Answer 1

Reset to default
3
$\begingroup$

There are plenty of examples of orthogonal designs for continuous predictors in the experimental design literature. A simple one is the design matrix (using centred predictors)

$$\boldsymbol{X}=(\boldsymbol{I},\boldsymbol{x}_1,\boldsymbol{x}_2)=\left(\begin{matrix} 1 & -1 & -1\\ 1 & -1 & 0\\ 1 & -1 & 1\\ 1 & 0 & -1\\ 1 & 0 & 0\\ 1 & 0 & 1\\ 1 & 1 & -1\\ 1 & 1 & 0\\ 1 & 1 & 1\\ \end{matrix}\right)$$

for the linear regression $$y_i=\beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} +\varepsilon_i$$

The diagonal variance–covariance matrix for the parameter estimates

$$\operatorname{Var} \boldsymbol{\hat\beta}= (\boldsymbol{X}^\mathrm{T}\boldsymbol{X})^{-1}\sigma^2=\left(\begin{matrix} \tfrac{1}{9} & 0 & 0\\ 0 & \tfrac{1}{6} & 0\\ 0 & 0 & \tfrac{1}{6}\\ \end{matrix}\right)\sigma^2$$

where $\sigma^2$ is the error variance, shows that you have uncorrelated estimators for $\beta_1$ & $\beta_2$

Improve this answer
$\endgroup$
4
  • $\begingroup$ It is clear that for this design matrix the inputs are orthogonal. As far as I understand, you centered predictors. That means you "precessed" the inputs in such a way to become orthogonalized. As a conclusion, I have to understand that enforcing means to design such kind of transformations in order to achieve orthogonality after I collected the data? So, the author does not mean by enforcing some precesses or criteria imposed before data collection, right? $\endgroup$
    – rapaio
    Jan 14, 2014 at 14:37
  • 1
    $\begingroup$ Centring is usually taken for granted when you talk about columns of the design matrix being orthogonal; the important thing is that the predictors' coefficient estimates are uncorrelated. If you don't centre (e.g. use $(1,2,3)$ for $(-1,0,1)$ in the above example), the estimates of $\beta_1$ & $\beta_2$ are the same, & the only elements that change in the variance-covariance matrix are those associated with the intercept. The authors do mean setting the predictors to pre-selected levels & observing the response. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jan 14, 2014 at 15:04
  • 1
    $\begingroup$ @Scortchi: is your last centred formula correct? There is an equality sign missing, and also, what is $\sigma$? Apart from that, what exactly do you mean by "uncorrelated estimates for $\beta_1$ & $\beta_2$"? Each $\beta$ is a number; two numbers cannot be correlated or not; what am I missing? $\endgroup$
    – amoeba
    Feb 8, 2014 at 21:09
  • $\begingroup$ @amoeba: Thanks & sorry for my sloppiness. The $\beta$'s are numbers; their estimators, the $\hat\beta$'s, are random variables. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Feb 8, 2014 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.