In other contexts, orthogonal means "at right angles" or "perpendicular".
What does orthogonal mean in a statistical context?
Thanks for any clarifications.
$\begingroup$
$\endgroup$
6
-
2$\begingroup$ Thanks for the question. I have asked a more general one: what is so common among all cases of orthogonality. I was also interested to know how does statistical independence satisfy this property? physics.stackexchange.com/questions/67506 $\endgroup$– ValCommented Jun 30, 2013 at 14:52
-
6$\begingroup$ I am surprised that none of the answers here mention that usually it is meant in the mathematical "linear algebra" sense of the word. For example, when we speak of an "orthogonal set of variables" usually it is meant that $X^{T}X=I$ for the matrix with the set of variables $X$. "orthonormal" is used as well. $\endgroup$– probabilityislogicCommented Sep 13, 2013 at 7:25
-
5$\begingroup$ @probability "Orthogonal" has meaning for a vector space with a quadratic form $Q$: two vectors $v$ and $w$ are orthogonal if and only if $Q(v,w)=0$. "Orthonormal" means in addition that $Q(v,v)=1=Q(w,w)$. Thus "orthogonal" and "orthonormal" are not synonymous, nor are they restricted to finite matrices. (E.g., $v$ and $w$ may be elements of a Hilbert space, such as the space of $L^2$ complex-valued functions on $\mathbb{R}^3$ used in classical quantum mechanics.) $\endgroup$– whuber ♦Commented Sep 13, 2013 at 15:02
-
$\begingroup$ This link might help to understand the (non)connection of orthogonality and correlation. alecospapadopoulos.wordpress.com/2014/08/16/… $\endgroup$– RBirkelbachCommented Nov 4, 2014 at 9:34
-
$\begingroup$ The growing collection of different (but correct) answers indicates this is a good CW thread. $\endgroup$– whuber ♦Commented Sep 13, 2016 at 15:45
|
Show 1 more comment
12 Answers
$\begingroup$
$\endgroup$
5
It means they [the random variables X,Y] are 'independent' to each other. Independent random variables are often considered to be at 'right angles' to each other, where by 'right angles' is meant that the inner product of the two is 0 (an equivalent condition from linear algebra).
For example on the X-Y plane the X and Y axis are said to be orthogonal because if a given point's x value changes, say going from (2,3) to (5,3), its y value remains the same (3), and vice versa. Hence the two variables are 'independent'.
See also Wikipedia's entries for Independence and Orthogonality
-
28$\begingroup$ Because the distinction between correlation and lack of dependence is important, equating orthogonality with independence is not a good thing to do. $\endgroup$– whuber ♦Commented Jun 20, 2011 at 20:26
-
$\begingroup$ Since neither OP nor answerer has been active for over a year, it's probably worth editing this to at least make it a clear answer. I've attempted that. $\endgroup$ Commented Nov 4, 2014 at 9:47
-
2$\begingroup$ One common counterexample to this within statistics is PCA vs. ICA, with PCA enforcing orthogonality and ICA maximizing independence. $\endgroup$– jonaCommented Nov 4, 2014 at 15:38
-
6$\begingroup$ To the moderators: It's a shame this good, and very popular question, is "stuck" with an answer that so many of think would be better demoted (current score -4). As both OP and answerer have not been active for over a year, perhaps the "accepted" check can be removed and the question be left "open". The more complete answers below speak for themselves. $\endgroup$ Commented Nov 5, 2014 at 0:45
-
6$\begingroup$ @Assad mods can't remove the OP's acceptance. That's the province of the OP. $\endgroup$– Glen_bCommented Oct 24, 2017 at 1:29
$\begingroup$
$\endgroup$
2
I can't make a comment because I don't have enough points, so I'm forced to speak my mind as an answer, please forgive me. From the little I know, I disagree with the selected answer by @crazyjoe because orthogonality is defined as
$$E[XY^{\star}] = 0$$
So:
If $Y=X^2$ with symmetric pdf they are dependent yet orthogonal.
If $Y=X^2$ but pdf zero for negative values, then they are dependent but not orthogonal.
Therefore, orthogonality does not imply independence.
$\begingroup$
$\endgroup$
1
If X and Y are independent then they are Orthogonal. But the converse is not true as pointed out by the clever example of user497804. For the exact definitions refer to
Orthogonal : Complex-valued random variables $C_1$ and $C_2$ are called orthogonal if they satisfy ${\rm cov}(C_1,C_2)=0$
(Pg 376, Probability and Random Processes by Geoffrey Grimmett and David Stirzaker)
Independent: The random variables $X$ and $Y$ are independent if and only if $F(x,y) = F_X(x)F_Y(y)$ for all $x,y \in \mathbb{R}$
which, for continuous random variables, is equivalent to requiring that $f(x,y) = f_X(x)f_Y(y)$
(Page 99, Probability and Random Processes by Geoffrey Grimmett and David Stirzaker)
-
$\begingroup$ The link above has all orthogonal vectors are linearly independent but not all linearly independet vectors are orthogonal... $\endgroup$– confusedCommented Jul 13, 2020 at 10:22
$\begingroup$
$\endgroup$
6
@Mien already provided an answer, and, as pointed out by @whuber, orthogonal means uncorrelated. However, I really wish people would provide some references. You might consider the following links helpful since they explain the concept of correlation from a geometric perspective.
-
1$\begingroup$ The second link explained everything I wanted to know. Thanks! :) $\endgroup$ Commented Jan 8, 2014 at 3:45
-
-
1$\begingroup$ @Bernd First two links are not working. $\endgroup$ Commented Jan 3, 2017 at 0:29
-
1$\begingroup$ @overwhelmed I'm guessing this is the article that second link was pointing to. $\endgroup$ Commented Feb 8, 2017 at 6:49
-
1$\begingroup$ I don't think, that "orthogonal means uncorrelated", and I don't see where @whuber did point that out $\endgroup$– AlexisCommented Feb 9, 2020 at 12:07
$\begingroup$
$\endgroup$
1
A NIST website (ref below) defines orthogonal as follows, "An experimental design is orthogonal if the effects of any factor balance out (sum to zero) across the effects of the other factors."
In statistical deisgn, I understand orthogonal to mean "not cofounded" or "not aliased". This is important when designing and analyzing your experiment if you want to make sure you can clearly identify different factors/treatments. If your designed experiment is not orthogonal, then it means you will not be able to completely separate the effects of different treatments. Thus you will need to conduct a follow up experiment to deconfound the effect. This would be called augmented deisgn or comparitive design.
Independence seems to be a poor word choice since its used in so many other aspects of design and analysis.
NIST Ref http://www.itl.nist.gov/div898/handbook/pri/section7/pri7.htm
-
4$\begingroup$ +1 for introducing an experimental design context. The word "orthogonal" deserves to be used here because it actually is exactly the same thing as the mathematical concept: the (column) vectors representing the factors in the experiment, considered as elements of a Euclidean space, will indeed be orthogonal (at right angles, with a zero dot product) in an orthogonal design. $\endgroup$– whuber ♦Commented Jun 11, 2015 at 19:14
$\begingroup$
$\endgroup$
7
It's most likely they mean 'unrelated' if they say 'orthogonal'; if two factors are orthogonal (e.g. in factor analysis), they are unrelated, their correlation is zero.
-
4$\begingroup$ The correlation coefficient is (or is naturally interpretable as) the cosine of an angle. When it is zero, what do you think the angle is? :-) Uncorrelated does not mean unrelated! $\endgroup$– whuber ♦Commented Jun 20, 2011 at 14:14
-
$\begingroup$ I'm not saying you're wrong, but could you give me an example of something that's uncorrelated and related; or vice versa? I'm not sure I understand the difference. $\endgroup$– MienCommented Jun 20, 2011 at 14:43
-
$\begingroup$ And yes, I know that that angle would be 90°. A right angle is orthogonal. $\endgroup$– MienCommented Jun 20, 2011 at 14:57
-
8$\begingroup$ Let $X$ be a random variable taking values in $\{-1,0,1\}$ with equal probability and let $Y=X^2$. The correlation between $X$ and $Y$ is $\rho_{X,Y}=0$, but clearly they are related: $Y$ is a function of $X$. $\endgroup$ Commented Jun 20, 2011 at 19:34
-
$\begingroup$ Ah yes, thank you. But the opposite isn't possible, is it (if there isn't a third variable or something similar)? $\endgroup$– MienCommented Jun 20, 2011 at 20:23
$\begingroup$
$\endgroup$
According to https://web.archive.org/web/20160705135417/http://terpconnect.umd.edu/~bmomen/BIOM621/LineardepCorrOrthogonal.pdf, linear independency is a necessary condition for orthogonality or uncorrelatedness. But there are finer distinctions, in particular, orthogonality is not uncorrelatedness.
$\begingroup$
$\endgroup$
I asked a similar question What is the relationship between orthogonality and the expectation of the product of RVs, and I reproduce the answer here. Although orthogonality is a concept from Linear Algebra, and it means that the dot-product of two vectors is zero, the term is sometimes loosely used in statistics and means non-correlation. If two random vectors are orthogonal, then their centralized counterpart are uncorrelated, because orthogonality (dot-product zero) implies non-correlation of the centralized random vectors (sometimes people say that orthogonality implies that the cross-moment is zero). Whenever we have two Random Vectors $(X,Y)$, we can always centralize them around their means to make their expectation to be zero. Assume ortogonality ($X\cdot Y=0$), then the correlation of the centralized random variables are $$Cov(X-E[X],Y-E[Y]) = E[X\cdot Y]= E[0]=0\implies \\Corr(X-E[X],Y-E[Y])=0$$
$\begingroup$
$\endgroup$
In econometrics, the orthogonality assumption means the expected value of the sum of all errors is 0. All variables of a regressor is orthogonal to their current error terms.
Mathematically, the orthogonality assumption is $E(x_{i}·ε_{i}) = 0$.
In simpler terms, it means a regressor is "perpendicular" to the error term.
$\begingroup$
$\endgroup$
Assume a random process x(t), hence y1=cos(x(t)) and y2= sin(x(t)), both are random processes. It is clear that y1 is orthogonal on y2, i.e., E[y1.y2] = 0. However, indeed they are dependent on each other. Actually, both are based on the same random process. Therefore, it is not necessary for orthogonal processes to be independent. Independence in random processes means that if you have any foreknowledge about one process, you will not be able to have any conclusion about the other! However, this is not the case with orthogonal processes. Nevertheless, assume two independent random processes z1, z2 where at least one of them has zero mean, then E[z1.z2]=E[z1].E[z2]=0. Mathematically, this is the same as the orthogonality condition, but geometrically, it is not necessary!
$\begingroup$
$\endgroup$
1
The related random variables mean the variables say X and Y can have any relationship; may be linear or non-linear. The independence and orthogonal properties are the same if the two variables are linearly related.
-
2$\begingroup$ This perpetuates the mistake made by crazyjoe and pat: orthogonality does not imply independence unless the variables are jointly normally distributed. $\endgroup$– whuber ♦Commented Jul 17, 2014 at 14:45
$\begingroup$
$\endgroup$
Two or more IV's unrelated (independent) to one another but both having an influence on the DV. Each IV separately contributes a distinct value to the outcome , while both or all IV's also contribute in an additive fashion in the prediction of income (orthogonal=non-intersecting IV's influence on a DV). IV's are non-correlational amongst one another and usually positioned in a right angle *see Venn Diagram.
Example: Relationship among motivation and years of education on income.
IV= Years of Education IV= Motivation DV= Income
https://web.archive.org/web/20160216160117/https://onlinecourses.science.psu.edu/stat505/node/167