# Can correlation matrix be used as features in machine learning classification

Can I use correlation between the training data as features, and if possible how will I test the test data with the model coefficients

I will try to explain more

If the training data are

X = [X1, X2,....., Xn]
and Xi = [Xi1,Xi2,....., Xi100]


where X are the training data and Xi is one sample of the data

and

K = [Xcorr1,1 ... Xcorr1,n
...      ...   ...
Xcorrn,1 ... Xcorrn,n]


K will be something like that

K = [1 .2 .3 .4
.2 1 .5 .6
.3 .5 1 .7
.4 .6 .7 1]


Can I will use this K to train my model ?

If it is possible How will I test my test data which will be

X = [X1, X2,....., Xn]

• If $K$ is the correlation matrix, why does your example have values outside the interval $[-1,1]$? Mar 23, 2022 at 17:24
• You use the language of machine learning, suggesting you are interested in predicting future responses. How would you construct a correlation matrix for such a case??
– whuber
Mar 23, 2022 at 17:38
• Isn't this what a [normalised] linear kernel does, e.g. for a support vector machine? If so, it will give the same behaviour as a conventional linear regression model on the original features (at least the unnormalised version) Mar 23, 2022 at 17:40
• @MahmoudReda I may have misunderstood what correlation function you were talking about. Would you mind giving an explicit mathematical expression in your question to elucidate? Mar 23, 2022 at 18:08
• "each value in the K matrix will be the weighted sum of the cross correlation between signal i and signal j " without knowing how and why they are weighted, I don't think it is possible to answer the question (note you don't appear to have mentioned that step in the question). It would really help to know why you are trying to construct features in this way. Mar 23, 2022 at 18:21

I think I may now understand the proposed algorithm, and it seems reasonable, but I don't think it is the way I would go about it.

The correlation in question is not the usual Pearson correlation, but the sort used in signal processing, which is related to convolutions and autocorrelations:

This function describes the similarities between the signals at different spatial distances.

As I understand it, the idea is to compute the correlation functions between the test signal and each of the training signals and compute a weighted sum, which will hopefully (if the genetic algorithm worked well) would pick out parts of the correlation function that contain discriminative information. So the feature is actually a similarity metric based on the correlation function.

This means the features of the new dataset are measures of the similarity between the test signals and each of the training signals. To generate the features for the test cases, it should be sufficient to just compute the covariance function between each test signal and each training signal and compute the weighted sums as before.

However, I think there is a much simpler approach. Just compute the covariance function between the signal and all of the training signals and just concatenate them together to make one very long vector and then use that as the input to a support vector machine. The performance bounds for the SVM don't depend on the dimensionality of the input vector, so provided you tune the regularisation parameter properly, over-fitting shouldn't be a problem and the SVM will identify the correct weightings for each lag in the correlation functions for you. As you can use the dual formulation of the SVM, there will be one parameter per training signal, rather than one for each element of the feature vector, which in this case will be reasonably efficient. This is likely to be much more effective than a GA, where there is little or no control over over-fitting.

• You got my idea very very well, And I will take your proposed idea into consideration, But I have a question - > for the cross-correlation approach what should be the label of the cross-correlation between a signal from class 1 and a signal from class 2? And another one of I want to classify an unknown signal with the cross-correlation approach should I cross-correlate it with some of my training signals then test it with the svm model Mar 23, 2022 at 22:10
• The label should be the class of the signal that is compared to each of the training samples. To get the input vector for the test signals, you just do exactly what you did for the training signals, i.e. compute all of the covariance functions and concatenate them together. It really is quite like a kernel method, look up "empirical kernel map", where a similar idea is used. Mar 23, 2022 at 22:16
• I really would avoid the GA though, I think it is likely to be a recipe for overfitting. Mar 23, 2022 at 22:17
• Thank you, Now I understand it very well. Mar 23, 2022 at 22:23

Of course can and should are different things. Of course you can take either the upper or lower triangle of the correlation matrix and train a machine learning model that takes them as input. While opinions, first principles, and guesswork might be helpful/misleading, the should part will become apparent from the performance of the models you try on such a feature space.

Using such a feature space as the correlations will average out lots of details about the original dataset. And correlation is itself translation-invariant and absolute-scale invariant, which also loses information, but z-score standardization is common (and often useful) anyway.

If you are predicting some variables from the correlation matrix in the training set, you will have to compute the corresponding correlation matrix for the test set as well.

My advice is to do a few tests to see how well it works.

The OP has clarified that they are not computing the usual Pearson correlation matrix, but rather a cross correlation. I quickly cooked up an example with correlation in the usual sense. For brevity I omit otherwise important aspects of machine learning such a cross validation. I may return to give a cross-correlation example in the future.

First import some modules and set a seed value for reproducibility.

import numpy as np
from sklearn.svm import SVC

np.random.seed(0)



Then let's define some population covariance matrices for three classes. I later assume that the mean vectors are the zero vector since the Pearson correlations are translation invariant anyway.

cov1 = np.eye(3)
cov2 = np.ones((3,3))
cov3 = [[1,-1,0],
[-1,1,-1],
[0,-1,1]]

covs = [cov1, cov2, cov3]


Now let's construct a $$3000 \times 3$$ matrix (1000 per population) whose rows are the $$m=10$$ samples and whose columns are the off-diagonal correlations. The sampling assumes a trivariate Gaussian distribution for each population. I apologize that the code is needlessly complicated due to stream-of-consciousness coding.

sample_corrs = []

k = 1000
m = 10

for i in range(k):
sample = [np.random.multivariate_normal(
np.zeros(3),
cov=covi,
size=10
) for covi in covs]

sample_corrs += [np.corrcoef(X_i.T)[np.triu_indices(3, 1)] for X_i in sample]

y = np.array([0,1,2] * k)
sample_corrs = np.array(sample_corrs)


Then train a support vector machine classifier (using a radial basis function as the kernel by default).

model = SVC()
model.fit(sample_corrs, y)


And check the accuracy on the training set.

>>> print(np.mean(model.predict(sample_corrs) == y))
0.934


So it appears you can discriminate populations based on their sample correlation matrices, at least in this idealized example.

I'll skip doing the cross-validation and hyperparameter tuning here since my goal here was just to show that some sort of model could be trained on a correlation matrices to predict a class.

Performing dimensionality reduction via principal component analysis on the $$3000 \times 3$$ matrix gives the following projection in 2D.

• I am assuming the random variables are non-degenerate. Degenerate variables will have not predictive value, and so should be excluded from predictive models. Mar 23, 2022 at 17:26
• Since a correlation characterizes an entire dataset, rather than individual cases, how do you propose to employ it for predictions or model fitting of any kind?
– whuber
Mar 23, 2022 at 17:39
• @whuber I was just contemplating that before you commented. Indeed, multiple datasets are needed to obtain multiple correlation matrices. This can be done by (a) finding multiple datasets or (b) creating multiple datasets. The latter can come about via partitioning an original dataset by an equivalence relation or by various simulated sampling methods. Mar 23, 2022 at 17:45
• @whuber I share your concerns with predicting a single observation from a sample aggregate such as a correlation vector. It would be clearer to construct a model that predicts a quantity paired to the correlation matrix (i.e. a sample) rather than an individual (i.e. group properties predicting group properties). If correlation matrices are always computed within a class, then you could try to predict the class that a correlation matrix belongs to. But I don't find this remotely as promising or as clear as predicting individual class membership from individual data. Mar 23, 2022 at 19:21

I don't see it, unless you have something like multiple multivariate time series and take the correlation matrix of the univariate time series (think multiple speech signals that you want to classify as being English or Spanish). (Let's ignore the time dependence and say that we're just going to slug through the calculation.)

In such a situation, that could be a feature extraction method. For each time series, you reduce the time series to a matrix, and then you use the matrix entries to predict the outcome. This even could result in different-length time series having the same number of features. I have my doubts that this will preserve the valuable information needed to make accurate predictions, but maybe there could be situations where this works.

However, if you just have a regular data frame (such as all of the time series concatenated together) and take the covariance matrix, you will wind up with one correlation matrix and a bunch of outcomes.

set.seed(2022)
N <- 1000
p <- 10
X <- matrix(rnorm(N*p, 0, 1), N, p)
y <- X %*% rnorm(p, 0, 1) + rnorm(N, 0, 1)
features <- corr(X)
L <- lm(y ~ features)


You don't have feature values corresponding to each outcome, so this code fails.

• Indeed, whether it is time or some other random variable or parameter, there has to be some distinguishing context to have multiple correlation matrices. As you suggest, such a model would have to predict values paired to each correlation matrix rather than individual objects of measure. Mar 23, 2022 at 18:02