2
$\begingroup$

Sorry if my question is quite basic! What does it exactly mean?

Correlation coefficients are not additive!

$\endgroup$
8
  • $\begingroup$ Does it mean that they are not intuitively linear and can not be for example summed or averaged? $\endgroup$
    – Ehsan
    Jan 24, 2015 at 17:13
  • $\begingroup$ Additivity is a very special property, most things are non-additive. $\endgroup$ Jan 24, 2015 at 21:00
  • $\begingroup$ thank you for your comment. was my guess in the first comment right? $\endgroup$
    – Ehsan
    Jan 24, 2015 at 21:09
  • $\begingroup$ What do you want to do? If you have correlation coefficient between some $x,y$-pair estimated in many different groups, and you want some common estimation of the correlation, assuming it is really (theoreticall) the same in all the groups, you might average them without problems. But maybe you want to do something else? $\endgroup$ Jan 24, 2015 at 21:11
  • $\begingroup$ In what sense was 'additivity' being discussed? Can you give the context? $\endgroup$
    – Glen_b
    Jan 25, 2015 at 0:22

2 Answers 2

2
$\begingroup$

Perhaps it means that $\rho(x+y, u + v) \neq \rho(x, u) + \rho(y, v)$ ?

Let's explore it: $\rho(x + y , u + v) \propto \mathbb{E}[ (x+y-\mathbb{E}(x+y))^\intercal \; (u+v - \mathbb{E}(u+v))]$

whereas $\rho (x,u) \propto \mathbb{E}[(x-\mathbb{E}(x))^\intercal \; (u - \mathbb{E}(u))]$ and equivalently for $\rho(y,v)$.

We see that $\rho(x+y, u+v)$ contains product terms $(y \, u)$, $(y \, \mathbb{E}(u))$, $(x\, v)$ and $(x\, \mathbb{E}(v))$ that are not present in $\rho(x,u)+\rho(y,v)$, so the two can't be equal by linearity of the expectation operator.

$\endgroup$
2
$\begingroup$

Did you know that a correlation is similar to an inner product?

An inner product is a measure that allows one to quantify how similar two mathematical objects are. If we're talking about two criss-crossing lines, this amounts to measuring if they are collinear, perpendicular, or somewhere in between. And so, using this picture, you can see that an inner product allows at least a qualitative measure of the "angle" between two mathematical objects. At minimum, any inner product can tell us whether or not two objects are "perpendicular." If one uses a normalized inner product, then one can fully measure the angles between objects.

There might be a lot of terminology here, so an example will help.

A typical inner product in the normal Euclidean setting in 2D or 3D space is the dot product. Say you have two 3D vectors, A = (A[1],A[2],A[3]) and B=(B[1],B[2],B[3]). (The results below are applicable to 2D vectors by simply assuming that A[3]=0 and B[3]=0.) The dot product of these vectors can be written in two important ways, both of which will give you the same answer. Deciding on which identity you use simply depends on your preference, or possibly how you collected your data: did you measure rectilinear components, magnitudes and angles, or both?

(1) dot(A,B) = ΣΑ[i]B[i]
(2) dot(A,B) = |A||B|cos(θ)

Here we use the notation |A| to denote the norm (aka the length) of the vector A. The norm is defined as follows:

(3) norm(A) = sqrt(dot(A,A))
      = sqrt(|A||A|cos(0)) = |A|

An important observation here is that the inner product (a mixed measurement of magnitudes and angle) on the space naturally induces a definition of the norm (magnitude, or "length") on that space.

Furthermore, the equality in Eq(2) tells us that if we can define a normalized inner product rather easily:

(4) normdot(A,B) = dot(A,B) / (|A||B|)
      = cos(θ)

Having this normalized inner product, we can quantitatively compute the angle between any two vectors in this space:

(5) θ = arccos(normdot(A,B))
      = arccos(dot(A,B)/(|A||B|))

The important take-away here is that in these simple vector spaces, the notions of vector components, vector magnitudes, and angles between vectors are all intuitive. In fact, mathematically speaking, the generalized definitions of inner product and norm was motivated by the desire to generalize these geometric notions to other mathematical spaces. Furthermore, these equations work in any finite-dimensional Euclidean space, no matter how high-dimensional the space is. Although we might not be able to picture it in our mind, given two N-dimensional vectors (A=(A[1],...,A[N]), B=(B[1],...,B[N])), we mathematically have the quantified notions of their magnitude, their components, and the angle between them.

Ok, so we're almost at the point of understanding why correlation coefficients are non-additive. We'll get there by imagining that we have two ordered N-element, mean-subtracted data sets, A = (A[1], A[2], ..., A[N]) and B = (B[1], B[2], ..., B[N]).

It already looks like an N-dimensional Euclidean vector doesn't it? Given our data sets are composed of real numbers, all of the equations we worked out above are immediately applicable to these N-elements data sequences. However, the dot product is not necessarily an inner product here since our N-element data sequences do not necessarily belong to a vector space --- we'll discuss this a bit more at the end.

To relate this all to correlation, note that the sample covariance of an ordered, mean-subtracted data set is written: (1)* Cov(A,B) = (1/(N-1))ΣΑ[i]B[i]

The standard deviation is just the square root of the variance, which is just a special case of covariance: (3)* StDev(A) = sqrt(Var(A)) = sqrt(Cov(A,A))
      = (1/(N-1))ΣA[i]A[i]

The correlation coefficient has the following definition:
(4)* ρ(A,B) = Cov(A,B) / (stdDev(A)stdDev(B))

Note that you can then define the angle between two data sets by allowing Eq(2) to generalize to this situation:
(2)* Cov(A,B) = StdDev(A)StdDev(B)cos(θ)
(5)* θ = arcCos(Cov(A,B)/(stdDev(A)stdDev(B))

Given this generalization of Eq(2) to the statistical setting, you can see that the correlation coefficient is not just similar to a cosine, but is literally a cosine:

             ρ(A,B) = Cov(A,B)/(stdDev(A)stdDev(B)) = cos(θ)

The above equations are marked as asterisked versions of the previous equations for a pretty obvious reason. Covariance looks suggestive of an inner product, doesn’t it? In fact, for a mean-subtracted data set, it is literally identical in appearance to a weighted N-dimensional dot product. Going with appearances, the standard deviation appears identical to a weighted Euclidean norm. Furthermore, by identifying the covariance and standard deviation as a weighted dot product and a weighted Euclidean norm, respectively, the correlation is then seen to look exactly like the normalized dot product--and thus to the cosine of an angle.

Identical in appearance, yes, but not necessarily identical: You see, an inner product on a real-valued vector space is characterized by linearity [f(aX+Z,Y)=af(X,Y)+f(Z,Y)], symmetry in its arguments [f(A,B)=f(B,A)], and positive-definiteness [f(X,Y) ≥ 0, the equality of which is reached only for X=Y=0].

Correlation is certainly symmetric in its arguments and positive definite. It even satisfies the scalar portion of the linearity property [f(aX,Y)=af(X,Y)]. On a case-by-case basis, if we can conjure up a useful or believable definition of vector addition for a data set, then correlation would meet all the requirements an inner product! In general, however, correlation is only similar to an inner product.

Without or without a notion of vector addition on your data space, correlation is similar enough to the Euclidean inner product in general to conceive of it as a cosine (or cosine-like) quantity---and this should give you an idea of why correlation coefficients are not additive since, in general, cosines are not additive: cos(θ) + cos(φ) does not equal cos(θ+φ).

An implication of this is that you cannot simply use the arithmetic mean as a measure of central tendency for correlations.

$\endgroup$
4
  • 1
    $\begingroup$ At the outset, by claiming that correlation "is a type of inner product," you depart from the standard mathematical and statistical definition of the term. This has the potential to create such confusion that the subsequent discussion might not suffice to clear it up. $\endgroup$
    – whuber
    Oct 14, 2015 at 22:42
  • $\begingroup$ Hmm, I see your point. I had thought it was a good analogy, but it might not stretch far enough, huh? The correlation function easily satisfies 3/4 of the requirements: rescaling, symmetry in its arguments, and is positive definite. The trick is establishing a type of vector addition that makes sense such that it can satisfy condition #1 (linearity in its arguments). Do you think the analogy is salvageable? $\endgroup$ Oct 15, 2015 at 16:32
  • $\begingroup$ I think most of this post is fine--except for that analogy. $\endgroup$
    – whuber
    Oct 15, 2015 at 22:35
  • $\begingroup$ Thank you for pointing it out. I have adjusted this response accordingly. $\endgroup$ Oct 17, 2015 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.