How to calculate Pearson correlation coefficient of two 3-dimensional features? The book I read mentions:

Even though, single features do not correlate with each other, sets of features can correlate with other sets of features.

I have an accelerometer, gyroscope, and magnetometer data where all of which are 3-dimensional [x, y, z]. How to calculate if they correlate with each other (for example if accelerometer data correlate with gyroscope data)?
Edit: Adding the exact quote from the book per @whuber request.

However, this pre-processing step may not completely eliminate collinearity, since one or more of the predictors may be functions of two or more of the other predictors.


Second, the univariate importance approach will fail to identify groups of predictors that together have a strong relationship with the response. For example, two predictors may not be highly correlated with the response; however, their interaction may be.

 A: If you are willing to compare the correlation between pairs of vectors, say accelerometer vs gyroscope you could consider computing canonical correlations. The idea is very simple.
Suppose you have blocks $X = (X_1,X_2,X_3)$ (e.g. accelerometer data) and $Y = (Y_1, Y_2, Y_3)$ (gyroscope data), all have the same sample size $n$. Then you consider computing correlations between linear combinations of each, say
$$
r_{ab} = \text{cor}(a^\top X, b^\top Y),
$$
for some $a = (a_1,a_2,a_3)$ and $b = (b_1,b_2,b_3)$. The idea of canonical correlation analysis (CCA) is to look for those $a$ and $b$ which, upon certain constraints, lead to the highest $r_{ab}$ as possible.
If you denote by $S_{1}$ covariance matrix of $X$, $S_2$ the covariance matrix of $Y$ and by $S_{12}$ the cross-covariance of $X,Y$, then the CCA is computed by applying the singular value decomposition (SVD) to the matrix
$$C = S_1^{-1/2} S_{12} S_2^{-1/2}.$$
There can be at most $3$ highest correlations called canonical correlations and these are the singular values of $C$. The first canonical correlation is the most important since it gives the higher correlation possible between the two linear combinations. The optimal vectors $a,b$ are related to the eigenvectors of the SVD of $C$
With CCA you essentially build new (latent) variables by linearly combining the old ones and you try to measure the highest correlation possible between these new variables with the data at hand. If your original variables are not correlated the canonical correlations will be nearly zero. On the other hand, the higher the correlation between the original variables the higher will be the canonical correlations, at least the first canonical correlation.
Canonical correlation analysis can also be seen as a method for dimensionality reduction, e.g. a kind of principal component analysis but, when there are two blocks of variables. If you want to read more about check any textbook on multivariate statistics; e.g. Johnson and Whichern Applied Multivariate Statistical Analysis, Pearson, 6th edition, 2014, ISBN-13: 978-1292024943.
An R implementation of CCA is provided by the CCA package via the cc command or through the cancor command from the stats package.
I'm going to illustrate the use of CCA in a practical example using the dataset iris. In the iris dataset, there are two measures related to sepals, i.e. sepal width and sepal length and two measures related to the petals, petal width and petal length. Suppose we want to have an overall measure of the correlation between sepal measures and petal measures. To do this we run CCA as follows
library(CCA)
X <- iris[,1:2]
Y <- iris[,3:4]
mycca <- cc(X,Y)

> mycca$cor
[1] 0.9409690 0.1239369

The object mycca$cor contains the canonical correlations between the two linear combinations since we have two variables in each block. The first correlation is very high thus we can tell that sepal measures are highly correlated with petal measures.
One might ask if this value is statistically different from zero. To address we can compute a log-likelihood ratio test (again for full details check the aforementioned reference).
rho_hat <- mycca$cor
p = ncol(X)
q = ncol(Y)
n = nrow(X)

# log-likelihood ratio statistic
(w_b = -(n-1-0.5*(p+q+1))*sum(log(1-rho_hat^2)))
[1] 319.6608

# threshold: quantile of the appropriate 
# limiting distribution
> qchisq(p = 0.95, df = p*q)
[1] 9.487729

The test we are conducting here is for $H_0:$ "all canonical correlations are zero" against the alternative $H_1:$ "there is at least one non-zero canonical correlation". Since we reject $H_0$, we conclude that the first canonical correlation is different from zero.
A: The answer is canonical-correlation analysis (CCA) which is already given by @utobu. I am just leaving here, how I have done this in Python:
import numpy as np
from sklearn.cross_decomposition import CCA
from sklearn.preprocessing import StandardScaler

# Standardize with zero mean and unit variance.
scaler = StandardScaler() 
acc_sc = scaler.fit_transform(np_acc_rolled)
gyro_sc = scaler.fit_transform(np_gyro_rolled) 
mag_sc = scaler.fit_transform(np_mag_rolled) 

# Apply CCA. 
n_comp = 3
cca_1 = CCA(scale=False, n_components=n_comp)
cca_2 = CCA(scale=False, n_components=n_comp)
cca_3 = CCA(scale=False, n_components=n_comp)
cca_1.fit(acc_sc, gyro_sc)
cca_2.fit(acc_sc, mag_sc)
cca_3.fit(gyro_sc, mag_sc)
acc_c_1, gyro_c_1 = cca_1.transform(acc_sc, gyro_sc)
acc_c_2, mag_c_1 = cca_2.transform(acc_sc, mag_sc)
gyro_c_2, mag_c_2 = cca_3.transform(gyro_sc, mag_sc)

# Canonical variate correlations.
acc_gyro_corr = [np.corrcoef(acc_c_1[:, i], gyro_c_1[:, i])[1][0] for i in range(n_comp)]
acc_mag_corr = [np.corrcoef(acc_c_2[:, i], mag_c_1[:, i])[1][0] for i in range(n_comp)]
gyro_mag_corr = [np.corrcoef(gyro_c_2[:, i], mag_c_2[:, i])[1][0] for i in range(n_comp)]

print(f"Acc-Gyro cvc: {acc_gyro_corr}")
print(f"Acc-Mag cvc{acc_mag_corr}")
print(f"Gyro-Mag cvc{gyro_mag_corr}")

This prints:
Acc-Gyro cvc: [0.4856137213325398, 0.07371717790419037, 0.028344236059519964]
Acc-Mag cvc: [0.9696215850795505, 0.5802252927965852, 0.2706828800001764]
Gyro-Mag cvc: [0.4117381986036376, 0.3043019981940091, 0.07602987575700093]

So, this confirms that the accelerometer and magnetometer data I have highly correlate.
