In other contexts, orthogonal means "at right angles" or "perpendicular".

What does orthogonal mean in a statistical context?

Thanks for any clarifications.

  • 2
    Thanks for the question. I have asked a more general one: what is so common among all cases of orthogonality. I was also interested to know how does statistical independence satisfy this property? physics.stackexchange.com/questions/67506 – Val Jun 30 '13 at 14:52
  • 5
    I am surprised that none of the answers here mention that usually it is meant in the mathematical "linear algebra" sense of the word. For example, when we speak of an "orthogonal set of variables" usually it is meant that $X^{T}X=I$ for the matrix with the set of variables $X$. "orthonormal" is used as well. – probabilityislogic Sep 13 '13 at 7:25
  • 3
    @probability "Orthogonal" has meaning for a vector space with a quadratic form $Q$: two vectors $v$ and $w$ are orthogonal if and only if $Q(v,w)=0$. "Orthonormal" means in addition that $Q(v,v)=1=Q(w,w)$. Thus "orthogonal" and "orthonormal" are not synonymous, nor are they restricted to finite matrices. (E.g., $v$ and $w$ may be elements of a Hilbert space, such as the space of $L^2$ complex-valued functions on $\mathbb{R}^3$ used in classical quantum mechanics.) – whuber Sep 13 '13 at 15:02
  • This link might help to understand the (non)connection of orthogonality and correlation. alecospapadopoulos.wordpress.com/2014/08/16/… – RBirkelbach Nov 4 '14 at 9:34
  • The growing collection of different (but correct) answers indicates this is a good CW thread. – whuber Sep 13 '16 at 15:45

11 Answers 11

up vote -14 down vote accepted

It means they [the random variables X,Y] are 'independent' to each other. Independent random variables are often considered to be at 'right angles' to each other, where by 'right angles' is meant that the inner product of the two is 0 (an equivalent condition from linear algebra).

For example on the X-Y plane the X and Y axis are said to be orthogonal because if a given point's x value changes, say going from (2,3) to (5,3), its y value remains the same (3), and vice versa. Hence the two variables are 'independent'.

See also Wikipedia's entries for Independence and Orthogonality

  • 23
    Because the distinction between correlation and lack of dependence is important, equating orthogonality with independence is not a good thing to do. – whuber Jun 20 '11 at 20:26
  • Since neither OP nor answerer has been active for over a year, it's probably worth editing this to at least make it a clear answer. I've attempted that. – Assad Ebrahim Nov 4 '14 at 9:47
  • 1
    One common counterexample to this within statistics is PCA vs. ICA, with PCA enforcing orthogonality and ICA maximizing independence. – jona Nov 4 '14 at 15:38
  • 5
    To the moderators: It's a shame this good, and very popular question, is "stuck" with an answer that so many of think would be better demoted (current score -4). As both OP and answerer have not been active for over a year, perhaps the "accepted" check can be removed and the question be left "open". The more complete answers below speak for themselves. – Assad Ebrahim Nov 5 '14 at 0:45
  • @Assad mods cant remove the OP's acceptance. That's the province of the OP. – Glen_b Oct 24 '17 at 1:29

I can't make a comment because I don't have enough points, so I'm forced to speak my mind as an answer, please forgive me. From the little I know, I disagree with the selected answer by @crazyjoe because orthogonality is defined as

$$E[XY^{\star}] = 0$$

So:

If $Y=X^2$ with symmetric pdf they they are dependent yet orthogonal.

If $Y=X^2$ but pdf zero for negative values, then they dependent but not orthogonal.

Therefore, orthogonality does not imply independence.

  • 2
    What's the asterisk (star) in $ Y^{*} $? – mugen Nov 4 '14 at 16:58
  • 2
    @mugen, probably indicating the complex conjugate. – A. Donda Sep 8 '15 at 15:45
  • Note to self (and possibly to others) - I believe that $E[XY]$ (for real-valued functions we can do away with the complex conjugate(?)) is the inner product of the random variables $X$ and $Y$, defined as the expectation of the product of their pdf's: $\langle X,Y\rangle=E[XY]$ – Antoni Parellada Dec 2 '16 at 20:45

@Mien already provided an answer, and, as pointed out by @whuber, orthogonal means uncorrelated. However, I really wish people would provide some references. You might consider the following links helpful since they explain the concept of correlation from a geometric perspective.

  • 1
    The second link explained everything I wanted to know. Thanks! :) – Lenar Hoyt Jan 8 '14 at 3:45
  • Real-valued random variables X and Y are uncorrelated if and only if the centered variables X-E(X) and Y-E(Y) are orthogonal.[ref] – knedlsepp Jul 14 '15 at 19:38
  • 1
    @Bernd First two links are not working. – overwhelmed Jan 3 '17 at 0:29
  • @overwhelmed I'm guessing this is the article that second link was pointing to. – Josh O'Brien Feb 8 '17 at 6:49

If X and Y are independent then they are Orthogonal. But the converse is not true as pointed out by the clever example of user497804. For the exact definitions refer to

Orthogonal : Complex-valued random variables $C_1$ and $C_2$ are called orthogonal if they satisfy ${\rm cov}(C_1,C_2)=0$

(Pg 376, Probability and Random Processes by Geoffrey Grimmett and David Stirzaker)

Independent: The random variables $X$ and $Y$ are independent if and only if $F(x,y) = F_X(x)F_Y(y)$ for all $x,y \in \mathbb{R}$

which, for continuous random variables, is equivalent to requiring that $f(x,y) = f_X(x)f_Y(y)$

(Page 99, Probability and Random Processes by Geoffrey Grimmett and David Stirzaker)

A NIST website (ref below) defines orthogonal as follows, "An experimental design is orthogonal if the effects of any factor balance out (sum to zero) across the effects of the other factors."

In statistical deisgn, I understand orthogonal to mean "not cofounded" or "not aliased". This is important when designing and analyzing your experiment if you want to make sure you can clearly identify different factors/treatments. If your designed experiment is not orthogonal, then it means you will not be able to completely separate the effects of different treatments. Thus you will need to conduct a follow up experiment to deconfound the effect. This would be called augmented deisgn or comparitive design.

Independence seems to be a poor word choice since its used in so many other aspects of design and analysis.

NIST Ref http://www.itl.nist.gov/div898/handbook/pri/section7/pri7.htm

  • 2
    +1 for introducing an experimental design context. The word "orthogonal" deserves to be used here because it actually is exactly the same thing as the mathematical concept: the (column) vectors representing the factors in the experiment, considered as elements of a Euclidean space, will indeed be orthogonal (at right angles, with a zero dot product) in an orthogonal design. – whuber Jun 11 '15 at 19:14

It's most likely they mean 'unrelated' if they say 'orthogonal'; if two factors are orthogonal (e.g. in factor analysis), they are unrelated, their correlation is zero.

  • 3
    The correlation coefficient is (or is naturally interpretable as) the cosine of an angle. When it is zero, what do you think the angle is? :-) Uncorrelated does not mean unrelated! – whuber Jun 20 '11 at 14:14
  • I'm not saying you're wrong, but could you give me an example of something that's uncorrelated and related; or vice versa? I'm not sure I understand the difference. – Mien Jun 20 '11 at 14:43
  • And yes, I know that that angle would be 90°. A right angle is orthogonal. – Mien Jun 20 '11 at 14:57
  • 5
    Let $X$ be a random variable taking values in $\{-1,0,1\}$ with equal probability and let $Y=X^2$. The correlation between $X$ and $Y$ is $\rho_{X,Y}=0$, but clearly they are related: $Y$ is a function of $X$. – Max Jun 20 '11 at 19:34
  • Ah yes, thank you. But the opposite isn't possible, is it (if there isn't a third variable or something similar)? – Mien Jun 20 '11 at 20:23

According to http://terpconnect.umd.edu/~bmomen/BIOM621/LineardepCorrOrthogonal.pdf, linear independency is a necessary condition for orthogonality or uncorrelatedness. But there are finer distinctions, in particular, orthogonality is not uncorrelatedness.

I asked a similar question What is the relationship between orthogonality and the expectation of the product of RVs, and I reproduce the answer here. Although orthogonality is a concept from Linear Algebra, and it means that the dot-product of two vectors is zero, the term is sometimes loosely used in statistics and means non-correlation. If two random vectors are orthogonal, then their centralized counterpart are uncorrelated, because orthogonality (dot-product zero) implies non-correlation of the centralized random vectors (sometimes people say that orthogonality implies that the cross-moment is zero). Whenever we have two Random Vectors $(X,Y)$, we can always centralize them around their means to make their expectation to be zero. Assume ortogonality ($X\cdot Y=0$), then the correlation of the centralized random variables are $$Cov(X-E[X],Y-E[Y]) = E[X\cdot Y]= E[0]=0\implies \\Corr(X-E[X],Y-E[Y])=0$$

In econometrics, the orthogonality assumption means the expected value of the sum of all errors is 0. All variables of a regressor is orthogonal to their current error terms.

Mathematically, the orthogonality assumption is $E(x_{i}·ε_{i}) = 0$.

In simpler terms, it means a regressor is "perpendicular" to the error term.

Two or more IV's unrelated (independent) to one another but both having an influence on the DV. Each IV separately contributes a distinct value to the outcome , while both or all IV's also contribute in an additive fashion in the prediction of income (orthogonal=non-intersecting IV's influence on a DV). IV's are non-correlational amongst one another and usually positioned in a right angle *see Venn Diagram.

Example: Relationship among motivation and years of education on income.

IV= Years of Education IV= Motivation DV= Income

https://onlinecourses.science.psu.edu/stat505/node/167

The related random variables mean the variables say X and Y can have any relationship; may be linear or non-linear. The independence and orthogonal properties are the same if the two variables are linearly related.

  • 1
    This perpetuates the mistake made by crazyjoe: orthogonality does not imply independence unless the variables are jointly normally distributed. – whuber Jul 17 '14 at 14:45

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.