25
$\begingroup$

I've read an article saying that when using planned contrasts to find means that are different in an one way ANOVA, constrasts should be orthogonal so that they are uncorrelated and prevent the type I error from being inflated.

I don't understand why orthogonal would mean uncorrelated under any circumstances. I can't find a visual/intuitive explanation of that, so I tried to understand these articles/answers

https://www.psych.umn.edu/faculty/waller/classes/FA2010/Readings/rodgers.pdf

What does orthogonal mean in the context of statistics?

but to me, they contradict each other. The first says that if two variables are uncorrelated and/or orthogonal then they are linearly independent, but that the fact that they are linearly independant does not imply that they are uncorrelated and/or orthogonal.

Now on the second link there are answers that state things like "orthogonal means uncorrelated" and "If X and Y are independent then they are Orthogonal. But the converse is not true".

Another interesting comment in the second link state that the correlation coefficient between two variables is equal to the cosine of the angle between the two vectors corresponding to these variables, which implies that two orthogonal vectors are completely uncorrelated (which isn't what the first article claims).

So what's the true relationship between independence, orthogonal and correlation ? Maybe I missed something but I can't find out what it is.

$\endgroup$
  • 1
    $\begingroup$ Do none of the answers to the questions showing as "Linked" and "Related" to the right of this question satisfy you? $\endgroup$ – Dilip Sarwate Sep 6 '15 at 14:01
  • 2
    $\begingroup$ The two links I provided seem to provide solid answers but state different things, and when I look at related questions, I can see that people giving answers are far from agreeing with each other $\endgroup$ – Carl Levasseur Sep 6 '15 at 14:05
  • 2
    $\begingroup$ The confusion/perceived contradiction could be entirely due to the difference between linear independence and statistical independence. $\endgroup$ – jona Sep 6 '15 at 17:18
  • 1
    $\begingroup$ I think (ANOVA) constrasts should be orthogonal is a vital aspect of this question: this isn't just about random variables. There is also an extra emphasis on "independence" compared to the closely-related question that Xian suggested as a possible duplicate (in that question the OP stated they understood "independence" so that was largely taken for granted in the answers). So I suggest it isn't a duplicate, and second @jona that the confusion may well be wrapped up in the multiple meanings of "independence". $\endgroup$ – Silverfish Sep 6 '15 at 20:43
  • 1
    $\begingroup$ I also believe this is not a duplicate. That question does not refer to correlation, and the answer does not detail the possible difference between orthogonality and uncorrelatedness. Moreover, as the poster pointed out, there are contradicting answers given to different related questions. $\endgroup$ – A. Donda Sep 7 '15 at 0:10
39
$\begingroup$

Independence is a statistical concept. Two random variables $X$ and $Y$ are statistically independent if their joint distribution is the product of the marginal distributions, i.e. $$ f(x, y) = f(x) f(y) $$ if each variable has a density $f$, or more generally $$ F(x, y) = F(x) F(y) $$ where $F$ denotes each random variable's cumulative distribution function.

Correlation is a weaker but related statistical concept. The (Pearson) correlation of two random variables is the expectancy of the product of the standardized variables, i.e. $$ \newcommand{\E}{\mathbf E} \rho = \E \left [ \frac{X - \E[X]}{\sqrt{\E[(X - \E[X])^2]}} \frac{Y - \E[Y]}{\sqrt{\E[(Y - \E[Y])^2]}} \right ]. $$ The variables are uncorrelated if $\rho = 0$. It can be shown that two random variables that are independent are necessarily uncorrelated, but not vice versa.

Orthogonality is a concept that originated in geometry, and was generalized in linear algebra and related fields of mathematics. In linear algebra, orthogonality of two vectors $u$ and $v$ is defined in inner product spaces, i.e. vector spaces with an inner product $\langle u, v \rangle$, as the condition that $$ \langle u, v \rangle = 0. $$ The inner product can be defined in different ways (resulting in different inner product spaces). If the vectors are given in the form of sequences of numbers, $u = (u_1, u_2, \ldots u_n)$, then a typical choice is the dot product, $\langle u, v \rangle = \sum_{i = 1}^n u_i v_i$.


Orthogonality is therefore not a statistical concept per se, and the confusion you observe is likely due to different translations of the linear algebra concept to statistics:

a) Formally, a space of random variables can be considered as a vector space. It is then possible to define an inner product in that space, in different ways. One common choice is to define it as the covariance: $$ \langle X, Y \rangle = \mathrm{cov} (X, Y) = \E [ (X - \E[X]) (Y - \E[Y]) ]. $$ Since the correlation of two random variables is zero exactly if the covariance is zero, according to this definition uncorrelatedness is the same as orthogonality. (Another possibility is to define the inner product of random variables simply as the expectancy of the product.)

b) Not all the variables we consider in statistics are random variables. Especially in linear regression, we have independent variables which are not considered random but predefined. Independent variables are usually given as sequences of numbers, for which orthogonality is naturally defined by the dot product (see above). We can then investigate the statistical consequences of regression models where the independent variables are or are not orthogonal. In this context, orthogonality does not have a specifically statistical definition, and even more: it does not apply to random variables.

Addition responding to Silverfish's comment: Orthogonality is not only relevant with respect to the original regressors but also with respect to contrasts, because (sets of) simple contrasts (specified by contrast vectors) can be seen as transformations of the design matrix, i.e. the set of independent variables, into a new set of independent variables. Orthogonality for contrasts is defined via the dot product. If the original regressors are mutually orthogonal and one applies orthogonal contrasts, the new regressors are mutually orthogonal, too. This ensures that the set of contrasts can be seen as describing a decomposition of variance, e.g. into main effects and interactions, the idea underlying ANOVA.

Since according to variant a), uncorrelatedness and orthogonality are just different names for the same thing, in my opinion it is best to avoid using the term in that sense. If we want to talk about uncorrelatedness of random variables, let's just say so and not complicate matters by using another word with a different background and different implications. This also frees up the term orthogonality to be used according to variant b), which is highly useful especially in discussing multiple regression. And the other way around, we should avoid applying the term correlation to independent variables, since they are not random variables.


Rodgers et al.'s presentation is largely in line with this view, especially as they understand orthogonality to be distinct from uncorrelatedness. However, they do apply the term correlation to non-random variables (sequences of numbers). This only makes sense statistically with respect to the sample correlation coefficient $r$. I would still recommend to avoid this use of the term, unless the number sequence is considered as a sequence of realizations of a random variable.

I've scattered links to the answers to the two related questions throughout the above text, which should help you put them into the context of this answer.

$\endgroup$
  • 1
    $\begingroup$ +1 The distinctions you make here are very clear and helpful--I enjoyed reading the entire post. $\endgroup$ – whuber Sep 7 '15 at 16:32
  • $\begingroup$ +1 I liked how you wove together the other answers which could otherwise seem contradictory. Perhaps in part (b) it would be nice to mention something specifically about experimental design or ANOVA (since that was mentioned in the OP's question) - it's not immediately obvious, in the context of your answer, why "orthogonality" might be an interesting or indeed desirable property of an independent variable. $\endgroup$ – Silverfish Sep 7 '15 at 23:41
  • $\begingroup$ @Silverfish, you're right, I'll try to add that. $\endgroup$ – A. Donda Sep 8 '15 at 15:39
  • 2
    $\begingroup$ I beg to differ from whuber's laudatory comments. The definition of independence is dreadful: it seems to imply that random variables $X$ and $Y$ have the same cumulative probability distribution function (CDF or cdf) which is here denoted by $F(\cdot)$. And no, $F(x)$ and $F(y)$ do not denote the different CDFs of $X$ and $Y$. $F(\cdot)$ is a real-valued function of a real variable, and $F(x)$ and $F(y)$ denote the values of this function at the numbers $x$ and $y$. The correct phrasing would be $$F_{X,Y}(x,y) = F_X(x)F_Y(y)~\text{for all}~ x~\text{and}~y, -\infty < x,y < \infty.$$ $\endgroup$ – Dilip Sarwate Sep 10 '15 at 21:04
  • 2
    $\begingroup$ @DilipSarwate, puh-lease... $\endgroup$ – A. Donda Sep 10 '15 at 22:41
0
$\begingroup$

Here is my intuitive view: Stating that x and y are uncorrelated/orthogonal are both ways of saying that knowledge of the value of x or y does not enable a prediction of the other -- x and y are independent of each other -- assuming that any relationship is linear.

The correlation coefficient provides an indication of how well knowledge of x (or y) enables us to predict y (or x). Assuming linear relationships.

In a plane, a vector along the X axis can be varied in magnitude without changing its component along the Y axis -- the X and Y axes are orthogonal and the vector along X is orthogonal to any along Y. Varying the magnitude of a vector not along X, will cause both the X and Y components to vary. The vector is no longer orthogonal to Y.

If two variables are uncorrelated they are orthogonal and if two variables are orthogonal, they are uncorrelated. Correlation and orthogonality are simply different, though equivalent -- algebraic and geometric -- ways of expressing the notion of linear independence. As an analogy, consider the solution of a pair of linear equations in two variables by plotting (geometric) and by determinants (algebraic).

With respect to the linearity assumption -- let x be time, let y be a sine function. Over one period, x and y are both orthogonal and uncorrelated using the usual means for computing both. However knowledge of x enables us to predict y precisely. Linearity is a crucial aspect of correlation and orthogonality.

Though not part of the question, I note that correlation and non-orthogonality do not equate to causality. x and y can be correlated because they both have some, possibly hidden, dependence on a third variable. Consumption of ice-cream goes up in the summer, people go to the beach more often in the summer. The two are correlated, but neither "causes" the other. See https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation for more on this point.

$\endgroup$
0
$\begingroup$

Here is the relationship: If X and Y are uncorrelated, then X-E[X] is orthogonal to Y-E[Y].

Unlike that independent is a stronger concept of uncorrelated, i.e., independent will lead to uncorrelated, (non-)orthogonal and (un)correlated can happen at the same time. Example

I am being the TA of probability this semester, so I make a short video about Independence, Correlation, Orthogonality.

https://youtu.be/s5lCl3aQ_A4

Hope it helps.

$\endgroup$
  • $\begingroup$ This does not answer the question. $\endgroup$ – Michael R. Chernick Nov 13 '18 at 23:09
  • $\begingroup$ I revise the answer, hope this would help~@Michael Chernick $\endgroup$ – linan huang Nov 13 '18 at 23:21
  • $\begingroup$ @linanhuang People from Larx? $\endgroup$ – YHH Nov 14 '18 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.