3
$\begingroup$

In factor analysis, I believe, the correlation between two factors is measured by the cosine of degree between the factors.

For instance, the correlation between two factors below is cosine θ, the degrees between factor 1 and factor 2 (or simply cosine degrees between two axes).

enter image description here

I would appreciate any help!

$\endgroup$
5
  • 1
    $\begingroup$ Not very familiar with factor analysis, but according to its Wikipedia page, the expectation of each factor is $\mathbf{0}$, so this formulation definitely has something to do with the fact that $\text{Cov}(\mathbf{f}_1,\mathbf{f}_2) \equiv \mathbb{E}\sum_i(\mathbf{f}_1-\mathbb{E}\mathbf{f}_1)_i(\mathbf{f}_2-\mathbb{E}\mathbf{f}_2)_i = \mathbb{E}\sum_i{\mathbf{f}_1}_i{\mathbf{f}_2}_i=\mathbb{E}(\mathbf{f}_1\cdot\mathbf{f}_2)$ $\endgroup$ Jun 17, 2022 at 12:12
  • 2
    $\begingroup$ In analytic geometry this is the definition of the cosine! So, look to an account of analytic (coordinate) geometry for explanations. $\endgroup$
    – whuber
    Jun 17, 2022 at 13:08
  • $\begingroup$ Much appreciated! $\endgroup$
    – No Ru
    Jun 17, 2022 at 13:37
  • $\begingroup$ In the subject space, variables are vectors. If variables had been centeted (as usually in factor analysis), the cosine of the angle between them equals their correlation. The factors (or principal components) extracted - we usually draw them as axes (dimensions) of the latent space where the variables having produce them lie. But the factors are readily seen as (latent) variables too, i.e. they posess variance and so can be drawn as vectors (arrows) of a length, coinciding with those axes. As factors are centered variables too, their correlation = cosine between those axes. $\endgroup$
    – ttnphns
    Jul 12, 2022 at 23:07
  • $\begingroup$ Subject space stats.stackexchange.com/a/192637/3277. Variables as vectors and factors as axes (easy to turn into vectors) in the subject space stats.stackexchange.com/a/119758/3277. Pic in footnote 2 showing correlating factors "covering tightly" correlating variables stats.stackexchange.com/a/151688/3277 $\endgroup$
    – ttnphns
    Jul 12, 2022 at 23:15

1 Answer 1

4
$\begingroup$

There is a least squares explanation of this.

Let $y$ be one factor (it doesn't matter which) and $x$ be the other. Ordinary least squares regression of $y$ against $x$ (without an intercept!) gives the slope estimate

$$\hat\beta = \frac{y \cdot x}{|x|^2}$$

where $y\cdot x$ is the usual dot product (equal to the matrix product $y^\prime x$ if you like) and $|x|^2 = x\cdot x$ is the squared length of $x$ (assumed to be nonzero).

Thus, the least squares fit is

$$\hat y = \hat\beta x = \left(\frac{y \cdot x}{|x|^2}\right)x.$$

The vectors $y$ and $\hat y,$ along with their common origin, form a triangle. This is a right triangle with hypotenuse subtended by $y.$ Its right angle is at $\hat y.$ One way to tell is to use the fact that right angles are formed only between two orthogonal vectors: that is, their dot product is zero. So we compute the dot product of the two legs $y - \hat y$ (aka the residual) and $x:$

$$ (y - \hat y) \cdot x = y \cdot x - \left(\frac{y \cdot x}{|x|^2}\right)x \cdot x = y\cdot x - \left(\frac{y \cdot x}{|x|^2}\right)|x|^2 = y\cdot x - y\cdot x = 0.$$

(These are called the Normal Equations.)

The most elementary definition of the cosine of the angle between $x$ and $y$ is the ratio of the length of the leg $\hat y$ to the length of the hypotenuse $y:$

$$\cos \theta_{xy} = \frac{|\hat y|}{|y|} = \frac{\bigg|\frac{y \cdot x}{|x|^2}\bigg|\,|x|}{|y|} = \frac{|y\cdot x|}{|x||y|} = \frac{|y\cdot x| / n}{\sqrt{\left(\frac{|x|^2}{n}\frac{|y|^2}{n}\right)}} = \big|\rho(x,y)\big|,$$

the absolute value of the correlation coefficient, assuming $y$ and $x$ have been centered (that is, the components of each one sum to zero). $n$ is the common length of the vectors.

This is actually the unsigned version; by removing the absolute value around $y\cdot x$ we obtain its generalization to signed angles.

Most of the expressions appearing here have standard names in statistics. Here's a brief translation table. It continues to assume $x$ and $y$ are centered.

$$\begin{array}{ll} \text{Expression} & \text{Name} \\ \hline y & \text{Response or Dependent variable} \\ x & \text{Explanatory or Independent variable}\\ x\cdot (y - x\hat\beta)= 0 & \text{Normal equations for }\hat\beta\\ \hat y & \text{Fitted or predicted value}\\ y - \hat y & \text{Residual}\\ \hat \beta & \text{Estimated coefficient (slope)}\\ y\cdot x\,/\,n& \text{Covariance}\\ |y|^2\,/\,{n},\ |x|^2\,/\,n & \text{Variances}\\ |\hat y|^2\,/\,|y|^2& R^2,\text{ the coefficient of determination}\\ |\hat y|\,/\,|y| & |\rho|,\text{ the absolute Pearson correlation; also, the size of the cosine}\\ y\cdot x\,/\,(|y||x|) & \text{Correlation coefficient }\rho\\ \arccos\left(y\cdot x\,/\,(|y||x|)\right) & \text{(Signed) angle }\theta_{xy}\\ \hline \end{array}$$

$\endgroup$
1
  • $\begingroup$ Wows thanks for the detailed explanation. Much appreciated!! $\endgroup$
    – No Ru
    Jun 18, 2022 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.