# Why is the correlation between two coordinate axes measured by cosine?

In factor analysis, I believe, the correlation between two factors is measured by the cosine of degree between the factors.

For instance, the correlation between two factors below is cosine θ, the degrees between factor 1 and factor 2 (or simply cosine degrees between two axes).

I would appreciate any help!

• Not very familiar with factor analysis, but according to its Wikipedia page, the expectation of each factor is $\mathbf{0}$, so this formulation definitely has something to do with the fact that $\text{Cov}(\mathbf{f}_1,\mathbf{f}_2) \equiv \mathbb{E}\sum_i(\mathbf{f}_1-\mathbb{E}\mathbf{f}_1)_i(\mathbf{f}_2-\mathbb{E}\mathbf{f}_2)_i = \mathbb{E}\sum_i{\mathbf{f}_1}_i{\mathbf{f}_2}_i=\mathbb{E}(\mathbf{f}_1\cdot\mathbf{f}_2)$ Jun 17, 2022 at 12:12
• In analytic geometry this is the definition of the cosine! So, look to an account of analytic (coordinate) geometry for explanations.
– whuber
Jun 17, 2022 at 13:08
• Much appreciated! Jun 17, 2022 at 13:37
• In the subject space, variables are vectors. If variables had been centeted (as usually in factor analysis), the cosine of the angle between them equals their correlation. The factors (or principal components) extracted - we usually draw them as axes (dimensions) of the latent space where the variables having produce them lie. But the factors are readily seen as (latent) variables too, i.e. they posess variance and so can be drawn as vectors (arrows) of a length, coinciding with those axes. As factors are centered variables too, their correlation = cosine between those axes. Jul 12, 2022 at 23:07
• Subject space stats.stackexchange.com/a/192637/3277. Variables as vectors and factors as axes (easy to turn into vectors) in the subject space stats.stackexchange.com/a/119758/3277. Pic in footnote 2 showing correlating factors "covering tightly" correlating variables stats.stackexchange.com/a/151688/3277 Jul 12, 2022 at 23:15

There is a least squares explanation of this.

Let $$y$$ be one factor (it doesn't matter which) and $$x$$ be the other. Ordinary least squares regression of $$y$$ against $$x$$ (without an intercept!) gives the slope estimate

$$\hat\beta = \frac{y \cdot x}{|x|^2}$$

where $$y\cdot x$$ is the usual dot product (equal to the matrix product $$y^\prime x$$ if you like) and $$|x|^2 = x\cdot x$$ is the squared length of $$x$$ (assumed to be nonzero).

Thus, the least squares fit is

$$\hat y = \hat\beta x = \left(\frac{y \cdot x}{|x|^2}\right)x.$$

The vectors $$y$$ and $$\hat y,$$ along with their common origin, form a triangle. This is a right triangle with hypotenuse subtended by $$y.$$ Its right angle is at $$\hat y.$$ One way to tell is to use the fact that right angles are formed only between two orthogonal vectors: that is, their dot product is zero. So we compute the dot product of the two legs $$y - \hat y$$ (aka the residual) and $$x:$$

$$(y - \hat y) \cdot x = y \cdot x - \left(\frac{y \cdot x}{|x|^2}\right)x \cdot x = y\cdot x - \left(\frac{y \cdot x}{|x|^2}\right)|x|^2 = y\cdot x - y\cdot x = 0.$$

(These are called the Normal Equations.)

The most elementary definition of the cosine of the angle between $$x$$ and $$y$$ is the ratio of the length of the leg $$\hat y$$ to the length of the hypotenuse $$y:$$

$$\cos \theta_{xy} = \frac{|\hat y|}{|y|} = \frac{\bigg|\frac{y \cdot x}{|x|^2}\bigg|\,|x|}{|y|} = \frac{|y\cdot x|}{|x||y|} = \frac{|y\cdot x| / n}{\sqrt{\left(\frac{|x|^2}{n}\frac{|y|^2}{n}\right)}} = \big|\rho(x,y)\big|,$$

the absolute value of the correlation coefficient, assuming $$y$$ and $$x$$ have been centered (that is, the components of each one sum to zero). $$n$$ is the common length of the vectors.

This is actually the unsigned version; by removing the absolute value around $$y\cdot x$$ we obtain its generalization to signed angles.

Most of the expressions appearing here have standard names in statistics. Here's a brief translation table. It continues to assume $$x$$ and $$y$$ are centered.

$$\begin{array}{ll} \text{Expression} & \text{Name} \\ \hline y & \text{Response or Dependent variable} \\ x & \text{Explanatory or Independent variable}\\ x\cdot (y - x\hat\beta)= 0 & \text{Normal equations for }\hat\beta\\ \hat y & \text{Fitted or predicted value}\\ y - \hat y & \text{Residual}\\ \hat \beta & \text{Estimated coefficient (slope)}\\ y\cdot x\,/\,n& \text{Covariance}\\ |y|^2\,/\,{n},\ |x|^2\,/\,n & \text{Variances}\\ |\hat y|^2\,/\,|y|^2& R^2,\text{ the coefficient of determination}\\ |\hat y|\,/\,|y| & |\rho|,\text{ the absolute Pearson correlation; also, the size of the cosine}\\ y\cdot x\,/\,(|y||x|) & \text{Correlation coefficient }\rho\\ \arccos\left(y\cdot x\,/\,(|y||x|)\right) & \text{(Signed) angle }\theta_{xy}\\ \hline \end{array}$$

• Wows thanks for the detailed explanation. Much appreciated!! Jun 18, 2022 at 7:48