Why is correlation only defined between two variables?

Question

I am an MBA student is taking statistics courses.

Our statistics prof was teaching us about correlations in statistics. We learned about Pearson's Correlation of Coefficient which is defined as the correlation between only two variables.

I am asked the prof if the correlation can be calculated between more than two variables and he said "no". But I am struggling to understand why this is the case?

For example: Hurricanes are correlated with Wind Speed, Hurricanes are also correlated with Temperature - Although I am not a meteorologist don't actually know if this is true (I just assumed this to show a point), we can see that in theory more than two variables can be correlated.

Therefore, why do we only evaluate the correlation between two variables and not more than two variables?

Answer 1

Pearson correlation is defined as a measure of the linear relationship between two variables.

For other relationships, like multidimensional relationships, we use other names. For instance:

• one could use the eigenvalues of a principal component analysis to express a degree of correlation in a multivariate case.
• Another related concept is the variance inflation factor.
• In the case of the hurricanes example that you mention, one might also think of multiple correlation and this gives the $R^2$ value or coefficient of determination for a linear model where hurricanes are predicted by temperature and windspeed.

In addition there are constructs similar to Pearson correlation that use multiple variables. I have seen on this website before an expression like $E[(X-\mu_X)(Y-\mu_Y)(Z-\mu_Z)]/(\sigma_X\sigma_Y\sigma_Z)$. (but I can not find the original post)

Answer 2

Correlation is defined between more than two variables, through a correlation matrix. This is not a single number of course, but that is only natural given that it is describing correlation between several pairs of variables. This situation is analogous to many types of measurement in multivariate analysis, where we measure aspects of the behaviour by vectors or matrices. You might also be interested to note that the correlation matrix for a set of variables is sufficient to compute the coefficient-of-determination for any Gaussian linear regression involving those variables (see this related answer), so even when we extend our analysis to look at conditional correlations, the correlation matrix is sufficient for this purpose.

Of course, even with a correlation matrix, it is merely describing the pairwise correlation between each pair of variables and resulting linear relationships conditional on other variables. The reason these correlation values are for pairs of variables is that they are measuring the tendency for one thing to vary with respect to a second thing.

Answer 3

why do we only evaluate the correlation between two variables and not more than two variables?

It can be more than 2 variables. Three point correlation function (3PC) is used in cosmology, The Three-Point Correlation Function in Cosmology. It is formed for variables $x$, $y$ and $z$ with the following approximation with a constant factor of 1, $$3PC=Corr(x,y,z) = Corr(x,y) \cdot Corr(y,z) + Corr(y,z) \cdot Corr(z,x) + Corr(z,x) \cdot Corr(x,y)\cdot$$ It is cycling over. This could be extended to N-point correlation function.

Answer 4

Such a statistic would be hard to define and interpret. Say you have the variables $A$, $B$, and $C$. The pairwise correlation between $A$ and $B$ is close to $+1$ and the pairwise correlation between $B$ and $C$ is close to $-1$. What should the single number correlation between the three variables be? High, low, or maybe zero?

You could instead look at correlation matrices between all the variables, but it is a highly overrated approach that could be misleading, and lead to a false sense of understanding. One example is people blindly dropping from their analysis variables that are strongly correlated with each other, or weekly correlated with the dependent variable, missing the point that those relations may change when controlling for other variables. It is like in the parable about the blind men and an elephant:

A group of blind men heard that a strange animal, called an elephant, had been brought to the town, but none of them were aware of its shape and form. Out of curiosity, they said: "We must inspect and know it by touch, of which we are capable". So, they sought it out, and when they found it they groped about it. The first person, whose hand landed on the trunk, said, "This being is like a thick snake". For another one whose hand reached its ear, it seemed like a kind of fan. As for another person, whose hand was upon its leg, said, the elephant is a pillar like a tree-trunk. The blind man who placed his hand upon its side said the elephant, "is a wall". Another who felt its tail, described it as a rope. The last felt its tusk, stating the elephant is that which is hard, smooth and like a spear.

Looking at the pairwise relations tells you only the part of the story but is not enough to get the full picture right. We all hope that our brain would be able to somehow combine all the information for the full picture, but if that was the case that people can just eyeball the numbers to draw legitimate conclusions, we wouldn't need statistics.

For many variables, we use instead the multivariate models like linear regression or partial correlations that tell us about pairwise relations but corrected for the influence of other variables.

Answer 5

Correlations between multiple variables can be defined as a joint cumulant. In physics we call this a "connected correlation function". In statistics, one would call these quantities covariances instead of correlations as it's customary to normalize correlations so that they are between -1 and 1.

The connected correlation between $n$ variables is an effect that's due to all $n$ variables together that cannot be attributed to correlations between variables in proper subsets of the $n$ variables. Connected correlation functions are multilinear functions of their arguments.

The expectation value of any arbitrary product of random variables can be written as a sum of products of connected correlations involving all the ways one can partition the set of variables into disjoint subsets. This property yields a recursion for the joint correlation function, so it can be used to define it.

If we denote connected correlations as $C(X_1,X_2,\ldots,X_n)$, then we have: $$C(X) = E(X)\tag{1}$$ which follows trivially from the recursion. For two variables $X$ and $Y$ we have that the connected correlation $C(X,Y)$ is the covariance between $X$ and $Y$. This also follows easily from the recursion: $$ E(XY) = C(X,Y) + C(X) C(Y) $$ Using (1) we then get: $$ C(X,Y)=E(XY) - E(X) E(Y)\tag{2} $$

For 3 variables $X$, $Y$ and $Z$, the recursion is: $$ E(XYZ) = C(X,Y,Z) + C(X,Y) C(Z) + C(X,Z) C(Y)+C(Y,Z) C(X) + C(X)C(Y)C(Z) $$

Substituting the expression (1) and (2) for the connected correlations of one and two variables then yields:

$$C(X,Y,Z)=E(XYZ) - E(XY) E(Z) - E(XZ)E(Y) - E(YZ)E(X) + 2E(X)E(Y)E(Z)$$

Answer 6

Main Answer

Why is correlation only defined between two variables?

Your professor likely meant Pearson's correlation as presented in the standard material you are required to learn. It is a definition used in the context of conventional (or at least introductory) statistics, and is certainly defined between only two random variables in that context. But let us explore this question beyond your course.

Ancillary Answer

Like many things, there is a tapestry of historical events that will never be fully uncovered. Auguste Bravais was the first person I am aware of to calculate what we would now think of Pearson's correlation on a sample, but to him they were merely cosines of the angles between error vectors. A little later came Francis Galton who laboured on the intuition of "co-relation" and his tabular calculations (he was not a skilled mathematician, purportedly) and was an inspiration to Karl Pearson who developed the formalism we recognize today. Why did these men consider only consider correlation to be between pairs of variables? I don't know. Even more restricting, Francis Galton appears to have only considered positive correlation which may have been due to his interest in biometrics which (by chance) were positively correlated (e.g. height and weight).

Here are my speculations (not to be confused with fact).

• Pairwise comparisons may be more intuitive for many people (though perhaps not all).
• Related to the first point, much of mathematics is riddled with binary operations. While multiary algebras and other fascinating creatures live in the world universal algebras, it is not on most people's radar. People develop tools with what they know about.
• Pearson's correlation interoperates nicely with linear algebra, and linear algebra is itself computationally feasible on modest problems in a pre-PC era.
• A computational sea monster awaits those that stray too far from pairwise decomposability: exponential complexity. Once you have posited some multiary function $f$, you quickly run into the problem that for $n$ possible operands there are $2^n$ possible inputs to the function (if you include inputting an empty set of variables, sometimes taken to be $f(\emptyset)=0$ or $f(\emptyset)=1$ depending on context). Making a number of statistical estimates that grows exponentially with the number of variables quickly gets out of hand, and it can take considerably more effort to think about what selection of subsets of the powerset of variables are needed for your problem.
• The multivariate distribution is often-applicable, and in the context pairwise comparisons are enough. Want a multivariate normal distribution but only have a vector of IID standard normal variables? No problem. Just apply a linear transformation. Plus Isserlis' theorem tells us that the higher mixed moments are either zero or decomposable into pairwise mixed moments. Combine this with Proposition 7.1.3 from Athreya and Lahiri 2006 and you'll realize that correlation tells you everything about statistical dependence of join normal distributions.

To an extent we can also deny the claim that we don't consider correlations of multiple variables, although I will side step the argument about what the word "correlation" ought to denote. As exemplified by other answers here (+1 to all I have seen so far), we actually do consider multiary functions of random variables that might be considered (in some sense) "co-relation". Clearly these functions are different, and thus selecting among them should be informed by what it is you want to quantify.

Examples

One thing I like to think about is what data sets minimize or maximize such statistics. And one way to do that is to create examples. Let me share some with you.

While the math in many cases is not restricted to three variables, I will keep to three for now just so that 3D scatterplots can be easily utilized. The following were found using gradient-based minimization (RMSProp). I did not check the functions for the existence of local minima, and I only reported the value of the objective to four decimal places.

Three-Point Correlation

This notion is from msuzen's post, which was new to me (+1).

	Minimal	Maximal
Plot
Score	-1.0000	3.0000

The minimal data set appears to be 'nearly' a line, but with some jittering. The maximal data set is a line.

Coskewness

The coskewness is the standardized mixed product moment of three variables. It is a trilinear extension of the bilinear Pearson product-moment correlation coefficient. This notion was explicitly mentioned by Sextus, and is a standardized cumulant of the sort shown by Count. The tensorial aspect is nicely shown by whuber.

	Minimal	Maximal
Plot
Score	-1.0000	1.0000

Multiple lines stretch out from the centroid into specific octants depending on the signum of $x_i y_i z_i$. Reflections $-x_i y_i z_i$ are avoided because of the odd symmetry: $\mathbb{E}[XYZ] = -\mathbb{E}[XYZ] = 0$. This will always be true for odd mixed moments.

Partial Correlation

With partial correlation we are computing the correlation between the residuals of $X$ as a function of $Z$ and the residuals of $Y$ as a function of $Z$. I noticed Tim also mentioned this statistic.

While in some of my own projects I consider the correlations of residuals of non-linear functions to be "partial correlations", it seems this is idiosyncratic to me. So for convention and and clarity, let us use the following linear equations:

$$Y = Z - 2$$ $$X = -3Z + 4$$

	Minimal	Maximal
Plot
Score	-1.0000	1.0000

Appearances can be deceiving. In both plots the data superficially appear to be spherical blobs, but actually all of the data approximately sits on a plane. Looking colinear to these planes, the data would appear to just follow a line.

Taylor's Multi-Way Correlation Coefficient

Taylor 2020 suggested the coefficient

$$\operatorname{mcor}[\vec x_1, \cdots, \vec x_n] \triangleq \frac{1}{\sqrt{d}} \sqrt{\frac{1}{d-1} \sum_{i=1}^d (\lambda_i - \bar \lambda)^2}$$

where $\lambda_i$ is the $i$th eigenvalue of the correlation matrix on $\vec x_1, \cdots, \vec x_n$ and $\bar \lambda$ is the mean eigenvalue of the same correlation matrix. This approach echos Sextus in considering the eigenvalues of a covariance matrix.

	Minimal	Maximal
Plot
Score	9.7931e-04	1.0000

The minimal data set is a spherical cloud of points. The maximal data set is a line.

Wang-Zheng's Unsigned Correlation Coefficient

Wang & Zheng 2014 proposed

$$\operatorname{UCC}[X_1, \cdots, X_n] \triangleq 1 - \det R_{\vec x \vec x}$$

where $R_{\vec x \vec x}$ is the correlation matrix on those variables.

	Minimal	Maximal
Plot
Score	3.7551e-06	1.0000

The minimal data set appears to be a spherical cloud of points. The maximal cloud appears to be nearly a plane except for a couple deviating points.

Coefficient of Multiple Correlation ($R^2$)

Sextus mentioned the multiple correlation coefficient. As noted by dipetkov, we must assume something asymmetric:

	Minimal	Maximal
Plot
Score	6.8780e-20	1.0000

The minimal data set is a spherical point cloud. The maximal is a data set which sits on a plane, and appears somewhat bimodal in distribution.

Variance Inflation Factor

Sextus mentioned the variance inflation factor. As with multiple correlation, we assume that $Y$ is predicted linearly from $X$ and $Z$:

	Minimal	Maximal
Plot
Score	1.0000	Unbounded! I stopped at 71.1589.

The minimal data set is a spherical point cloud. In the maximal case it isn't really maximal, but rather unbounded. In this unbounded case the data appears to be approaching a distribution similar to what maximized the multiple correlation coefficient, but I stopped before numerical stability collapsed.

User1865345's Suggestion

User1865345 suggested we consider something of the form:

$$R[X,Y]R[X,Z]$$

	Minimal	Maximal
Plot
Score	-1.0000	1.0000

Both the minimal and maximal cases are data sets in the form of a line with a little bit of jitter.

Answer 7

One of the interpretations of Pearson's correlation coefficient is the proportionate reduction of error (PRE): Assume a variable vector $y$, about which nothing further is known. Then, if you have to predict a missing $y_i$, your best estimator would be the arithmetic mean $\bar{y}$, and the error would be $E_1 = SS_{total} = \sum_{i=1}^n(y_i - \bar{y})^2$.

Assume now that you have a second data set $x$, which can be used to predict $y$, in the simplest case (but without limitation of generality) by linear regression $\hat{y}_i = a + b x_i$. Then the error becomes $E_2 = SS_{residual} = \sum_{i=1}^n(y_i - \hat{y}_i)^2$, and the coefficient of determination (fraction of variability of $y$ that is explained by $x$) is $r^2 = \frac{E_1 - E_2}{E_1} = 1 - \frac{E_2}{E_1}$. The $R^2$ mentioned by others is a generalisation of this approach for several explaining variables. There are other correlation coefficients with PRE-interpretation for non-cardinal data (Guttman's λ, Freeman's θ, Wilson's e, the point-biserial correlation, $\eta^2$). See Freeman, L.C.: Elementary applied statistics, New York, London, Sidney (John Wiley and Sons) 1965 for a very readable introduction.

Thus, you should think about what PRE is in your particular case to construct a suitable correlation coefficient.