# Using canonical correlation analysis (CCA) to find matches

I have a training dataset of images: X (Visual) and Y (Infrared). Each set has $300$ training examples. I extract feature vectors from both sets of images. Thus my X and Y datasets are respectively $300\times 1920$ and $300\times 1536$ where $300$ is the number of sample observations (in this case images) and $1920$ & $1536$ are respectively the length of the feature vectors in visual and infrared spectrum.

A testing dataset consists of images of different subjects in both visual spectrum and infrared spectrum. I will have the visual spectrum data as my gallery. I will have the infrared images as the probe data. For each probe image, I need to retrieve the corresponding visual image from the gallery by using some kind of similarity measure.

Basic algorithm idea:

1. Start Training Phase. Read the images from visual and infrared spectrum dataset.
2. Get the feature vectors by using desired descriptors and populate the matrices X and Y.
3. Use CCA for subspace learning. Get the projection matrices Wx and Wy.
4. Start Testing Phase. Given the gallery images (visual), read these images and use the Wx transformation matrix to convert them into the CCA subspace.
5. For each probe image, convert it into the CCA subspace by using the Wy transformation and compare with each images of the gallery and compute a matching score.
6. The image (or its label) in the gallery having the maximum score is returned.

Could anyone tell me if my approach is correct or not? Pointing me in the right direction would also be helpful. Please see the following paper for reference: Yi et al. 2007, Face Matching Between Near Infrared and Visible Light Images.

I work in Matlab and use the following command to perform CCA:

[Wx,Wx,r,U,V] = canoncorr(X,Y);   %// DO CCA


The output I get is this :

  Name           Size             Bytes   Class     Attributes

Wx         1920x297            890880  double
Wx         1536x297            712704  double
U           300x297             27840  double
V           300x297             27840  double
r             1x297               464  double

1. The projection matrices are Wx and Wy since they transform X and Y into the new space.

2. The resulting projections of X and Y into the new space are U and V, respectively.

3. The r vector represents the entries of the correlation matrix between U and V, which is a diagonal matrix.

• Algebra and important matrices of CCA you can find here. Specifically for Matlab, please wait somebody knowing it to answer. – ttnphns Jan 28 '15 at 7:24
• thanks for pointing out that. Actually I have already checked out your visual aid for understanding CCA. it was very clear. My question is not pertinent to only MATLAB. Specifically I am asking that if I know the projection matrices A and B from the training data. Can I then use this on the testing data to find matches on the testing data? – roni Jan 28 '15 at 8:10
• Please tell me whether I should better rephrase my question ? – roni Jan 28 '15 at 8:11
• I edited your question such that it reads as follow-up one and not as a repetition of what is already answered on StackOverflow. Please check that it is correct. Also, I find the description of your task a little unclear; perhaps you could clarify it. – amoeba says Reinstate Monica Jan 28 '15 at 10:26
• I don't really understand your step #5. When you project your probe image onto CCA subspace, you get a vector of length $297$ (according to your Matlab output). When you project each gallery image onto the CCA subspace, you also get a vector of length $297$. How are you then going to compute the similarity score? Even if you only use one first CCA component, so that you have only one number instead of $297$-long vectors, it does not seem to be trivial to compute the similarity score; but with many components it's even less clear. – amoeba says Reinstate Monica Jan 28 '15 at 14:31

This looks like a possible approach.

CCA will find pairs of vectors $(\mathbf w, \mathbf v)$ such that projections $\mathbf X \mathbf w$ and $\mathbf Y \mathbf v$ have maximal possible correlations (the pairs will be ordered in the order of decreasing correlations). Projection vectors are normalized such that the variance of $\mathbf X \mathbf w$ and of $\mathbf Y \mathbf v$ is equal to $1$. This means that projections are not only correlated, but "on the same scale" and hence can be directly compared.

Some things to keep in mind is that: (1) you can only center your test data with the mean of the training data; (2) in high dimensions CCA is prone to overfitting and it will be a good idea either to use regularized CCA or to preprocess the data with PCA.

Here is a very simple Matlab script implementing this approach:

% // Using Fisher Iris data.
% // X will be petal measurements, Y will be sepal measurements
trainN = 75; %// using half of the data for training and half for testing

centerTrain = mean(meas(1:trainN,:));
X = bsxfun(@minus, meas(:,1:2), centerTrain(1:2));
Y = bsxfun(@minus, meas(:,3:4), centerTrain(3:4));

% // This computes CCA on the training data
[A,B,r] = canoncorr(X(1:trainN,:), Y(1:trainN,:));

% // Projecting the test data
Xtestpr = X(trainN+1:end, :) * A;
Ytestpr = Y(trainN+1:end, :) * B;

% // Loop over all train samples
correct = 0;
for i=1:size(Xtestpr,1)
% // Using only the first CCA projection, find the sample in Y
% // closest to the one in X. Euclidean distance is used as a
% // similarity measure.
[~, ind] = min(sum((Xtestpr(i,1) - Ytestpr(:,1)).^2, 2));

% // if classified correctly
if ind==i
correct = correct+1;
end
end

%// compute the probability that so many correct matchings
%// could be obtained by chance
pval = 1 - binocdf(correct-1, size(Xtestpr,1), 1/size(Xtestpr,1));

%// compute confidence interval on correct matching rate
[~, cinf] = binofit(correct, size(Xtestpr,1));


This gives me $6$ correct classifications out of $75$, which does not sound like a lot, but still is clearly significant with a p-value of $0.0005$. Confidence interval on matching probability is $(0.03, 0.17)$.

• Thanks for this skeletal code. I am looking into it. I will let you know. – roni Feb 11 '15 at 16:33
• I ran the code. It looks fine. I also got 6 matches. Can you tell me why is still significant as you pointed out in the text? and I am sorry but could you explain a bit about p-value ? Can p-value be used to get a rough estimate of the recognition rate ? – roni Feb 11 '15 at 17:39
• Well, we get 6 matches out of 75. P-value is the probability that we could have obtained so many (or even more!) matches by pure guessing. The probability to get a correct match by chance is 1/75 for each sample. One can compute the probability to get 6 or more matches with probability 1/75, it's equal to 0.0005. This is a very low number, so we can argue that getting 6 matches by pure chance is very improbable, and hence our algorithm is better than guessing. But note that in this case I only used 1 CCA direction; in real applications you will use more and hopefully get better performance. – amoeba says Reinstate Monica Feb 11 '15 at 17:46
• If you want a recognition rate, you can compute a 95% confidence interval on the probability of success given 6 successes out of 75. In Matlab, write [~,int]=binofit(6,75), this gives an interval (0.03, 0.17). That's the interval where the recognition rate is likely to be. Note that this interval does not include 1/75=0.013, which also indicated that it's significant. – amoeba says Reinstate Monica Feb 11 '15 at 17:48
• Well, that's because the second CCA dimension is not really informative. If you run get correlations from canoncorr, you will see that they are equal to 0.93 and 0.08, so presumably the second CCA dimension does not carry much information. In real-life application you would probably want to optimize how many components you use by cross-validation. – amoeba says Reinstate Monica Feb 11 '15 at 17:59